2023-10-30T11:49:03.761 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-10-30T11:49:03.762 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind} --verbose --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070 --name amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070/orig.config.yaml 2023-10-30T11:49:03.798 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T11:49:03.877 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7441070' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: cc36dd85d142c6fdb77515cad8af108cec98a86d valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: debuginfo: true flavor: default sha1: cc36dd85d142c6fdb77515cad8af108cec98a86d thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: d0db9f2f95b9d6344950185071cace2a50d296eb owner: scheduled_amathuri@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 1876 sha1: cc36dd85d142c6fdb77515cad8af108cec98a86d sleep_before_teardown: 0 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d0db9f2f95b9d6344950185071cace2a50d296eb targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB0aWaga+dsDrTwqHzQWFfUjBQaami3eA1PWTEDHnqNfmchWzCWJ7rBcGnS6wHE8cDcDGGiFpluElQlxZf8rN+w= smithi092.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBABPeEmKQ4kin+AOWQHfOC8gZ7lIz1qIaXii2x99f4NDEOM4MA9xORRfqdfj44euxLjUEZy/r5gm4sjfbgrWH7E= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-30_10:36:20 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-10-30T11:49:03.878 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa; will attempt to use it 2023-10-30T11:49:03.879 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks 2023-10-30T11:49:03.879 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-30T11:49:03.881 INFO:teuthology.task.internal:Checking packages... 2023-10-30T11:49:03.904 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'cc36dd85d142c6fdb77515cad8af108cec98a86d' 2023-10-30T11:49:03.904 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-30T11:49:03.905 INFO:teuthology.packaging:ref: None 2023-10-30T11:49:03.905 INFO:teuthology.packaging:tag: None 2023-10-30T11:49:03.905 INFO:teuthology.packaging:branch: main 2023-10-30T11:49:03.906 INFO:teuthology.packaging:sha1: cc36dd85d142c6fdb77515cad8af108cec98a86d 2023-10-30T11:49:03.906 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2023-10-30T11:49:04.219 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6863.g636384c4 2023-10-30T11:49:04.219 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-30T11:49:04.226 INFO:teuthology.task.internal:no buildpackages task found 2023-10-30T11:49:04.226 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-30T11:49:04.248 INFO:teuthology.task.internal:Saving configuration 2023-10-30T11:49:04.262 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-30T11:49:04.268 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T11:49:04.294 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 11:44:05.001677', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB0aWaga+dsDrTwqHzQWFfUjBQaami3eA1PWTEDHnqNfmchWzCWJ7rBcGnS6wHE8cDcDGGiFpluElQlxZf8rN+w='} 2023-10-30T11:49:04.315 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 11:44:05.003252', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBABPeEmKQ4kin+AOWQHfOC8gZ7lIz1qIaXii2x99f4NDEOM4MA9xORRfqdfj44euxLjUEZy/r5gm4sjfbgrWH7E='} 2023-10-30T11:49:04.316 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-30T11:49:04.322 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-10-30T11:49:04.322 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-10-30T11:49:04.323 INFO:teuthology.run_tasks:Running task console_log... 2023-10-30T11:49:04.393 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f50db32da60>, signals=[15]) 2023-10-30T11:49:04.394 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-30T11:49:04.401 INFO:teuthology.task.internal:Opening connections... 2023-10-30T11:49:04.401 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2023-10-30T11:49:04.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:49:04.500 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2023-10-30T11:49:04.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:49:04.583 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-30T11:49:04.590 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2023-10-30T11:49:04.608 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2023-10-30T11:49:04.608 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2023-10-30T11:49:04.664 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream" 2023-10-30T11:49:04.665 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="9" 2023-10-30T11:49:04.665 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2023-10-30T11:49:04.665 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2023-10-30T11:49:04.665 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="9" 2023-10-30T11:49:04.666 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el9" 2023-10-30T11:49:04.666 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-30T11:49:04.666 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2023-10-30T11:49:04.666 INFO:teuthology.orchestra.run.smithi003.stdout:LOGO="fedora-logo-icon" 2023-10-30T11:49:04.667 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-30T11:49:04.667 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2023-10-30T11:49:04.667 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-30T11:49:04.667 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-30T11:49:04.667 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-30T11:49:04.668 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2023-10-30T11:49:04.693 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2023-10-30T11:49:04.711 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2023-10-30T11:49:04.711 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2023-10-30T11:49:04.768 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="CentOS Stream" 2023-10-30T11:49:04.769 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="9" 2023-10-30T11:49:04.769 INFO:teuthology.orchestra.run.smithi092.stdout:ID="centos" 2023-10-30T11:49:04.769 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="rhel fedora" 2023-10-30T11:49:04.769 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="9" 2023-10-30T11:49:04.770 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el9" 2023-10-30T11:49:04.770 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-30T11:49:04.770 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31" 2023-10-30T11:49:04.770 INFO:teuthology.orchestra.run.smithi092.stdout:LOGO="fedora-logo-icon" 2023-10-30T11:49:04.771 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-30T11:49:04.771 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://centos.org/" 2023-10-30T11:49:04.771 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-30T11:49:04.771 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-30T11:49:04.771 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-30T11:49:04.772 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2023-10-30T11:49:04.793 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-30T11:49:04.802 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-30T11:49:04.807 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-30T11:49:04.808 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2023-10-30T11:49:04.811 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2023-10-30T11:49:04.827 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-30T11:49:04.833 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-30T11:49:04.833 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2023-10-30T11:49:04.868 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2023-10-30T11:49:04.978 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-30T11:49:05.033 INFO:teuthology.run_tasks:Running task kernel... 2023-10-30T11:49:05.048 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-30T11:49:05.049 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-30T11:49:05.049 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T11:49:05.050 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T11:49:05.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T11:49:05.067 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2023-10-30T11:49:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T11:49:05.068 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2023-10-30T11:49:05.123 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-361.el9.x86_64 2023-10-30T11:49:05.123 INFO:teuthology.task.kernel:Running kernel on smithi092: 5.14.0-361.el9.x86_64 2023-10-30T11:49:05.124 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum install -y kernel 2023-10-30T11:49:05.126 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-361.el9.x86_64 2023-10-30T11:49:05.127 INFO:teuthology.task.kernel:Running kernel on smithi003: 5.14.0-361.el9.x86_64 2023-10-30T11:49:05.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2023-10-30T11:49:07.292 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 12 MB/s | 6.3 MB 00:00 2023-10-30T11:49:07.353 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - AppStream 11 MB/s | 6.3 MB 00:00 2023-10-30T11:49:08.651 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2023-10-30T11:49:08.662 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2023-10-30T11:49:09.810 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 7.9 MB 00:00 2023-10-30T11:49:10.192 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 7.9 MB 00:01 2023-10-30T11:49:12.444 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 11 MB/s | 18 MB 00:01 2023-10-30T11:49:18.036 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 5.7 MB 00:02 2023-10-30T11:49:18.583 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 18 MB 00:07 2023-10-30T11:49:19.553 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2023-10-30T11:49:21.205 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 13 MB/s | 19 MB 00:01 2023-10-30T11:49:25.150 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 1.9 MB/s | 5.7 MB 00:03 2023-10-30T11:49:25.654 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2023-10-30T11:49:26.509 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 15 kB 00:00 2023-10-30T11:49:27.677 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-30T11:49:27.678 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-30T11:49:27.772 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:49:27.784 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:49:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2023-10-30T11:49:27.785 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:49:27.785 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:49:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-10-30T11:49:27.786 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-10-30T11:49:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-10-30T11:49:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-10-30T11:49:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-10-30T11:49:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:49:27.787 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:49:27.787 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:49:27.787 INFO:teuthology.orchestra.run.smithi092.stdout:Install 4 Packages 2023-10-30T11:49:27.787 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:49:27.788 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 95 M 2023-10-30T11:49:27.788 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 120 M 2023-10-30T11:49:27.788 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:49:28.113 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 13 MB/s | 19 MB 00:01 2023-10-30T11:49:29.180 INFO:teuthology.orchestra.run.smithi092.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 4.9 MB/s | 5.3 MB 00:01 2023-10-30T11:49:30.030 INFO:teuthology.orchestra.run.smithi092.stdout:(2/4): kernel-core-5.14.0-378.el9.x86_64.rpm 10 MB/s | 20 MB 00:01 2023-10-30T11:49:30.872 INFO:teuthology.orchestra.run.smithi092.stdout:(3/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 13 MB/s | 38 MB 00:02 2023-10-30T11:49:31.389 INFO:teuthology.orchestra.run.smithi092.stdout:(4/4): kernel-modules-core-5.14.0-378.el9.x86_6 15 MB/s | 32 MB 00:02 2023-10-30T11:49:31.390 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:49:31.390 INFO:teuthology.orchestra.run.smithi092.stdout:Total 26 MB/s | 95 MB 00:03 2023-10-30T11:49:32.240 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:49:32.600 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:49:32.600 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:49:32.810 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2023-10-30T11:49:33.916 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:49:33.916 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:49:34.975 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-30T11:49:34.975 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-30T11:49:35.075 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:49:35.083 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:49:35.084 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-10-30T11:49:35.084 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:49:35.084 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:49:35.084 INFO:teuthology.orchestra.run.smithi003.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-10-30T11:49:35.085 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-30T11:49:35.085 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-10-30T11:49:35.085 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-10-30T11:49:35.085 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-10-30T11:49:35.085 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:49:35.086 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:49:35.086 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:49:35.086 INFO:teuthology.orchestra.run.smithi003.stdout:Install 4 Packages 2023-10-30T11:49:35.086 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:49:35.087 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 95 M 2023-10-30T11:49:35.087 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 120 M 2023-10-30T11:49:35.087 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:49:35.520 INFO:teuthology.orchestra.run.smithi003.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 20 MB/s | 5.3 MB 00:00 2023-10-30T11:49:35.854 INFO:teuthology.orchestra.run.smithi003.stdout:(2/4): kernel-core-5.14.0-378.el9.x86_64.rpm 33 MB/s | 20 MB 00:00 2023-10-30T11:49:36.371 INFO:teuthology.orchestra.run.smithi003.stdout:(3/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 34 MB/s | 38 MB 00:01 2023-10-30T11:49:36.519 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:49:36.671 INFO:teuthology.orchestra.run.smithi003.stdout:(4/4): kernel-modules-core-5.14.0-378.el9.x86_6 28 MB/s | 32 MB 00:01 2023-10-30T11:49:36.671 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:49:36.672 INFO:teuthology.orchestra.run.smithi003.stdout:Total 60 MB/s | 95 MB 00:01 2023-10-30T11:49:37.130 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-10-30T11:49:37.255 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:49:37.552 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:49:38.001 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:49:38.001 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:49:39.350 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:49:39.351 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:49:40.024 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:49:40.216 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:49:42.124 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:49:42.782 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-10-30T11:49:42.898 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:49:44.795 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:49:45.005 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:49:45.772 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:49:45.950 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:49:49.616 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:49:50.926 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:49:51.040 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:49:56.054 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:50:18.389 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:50:18.431 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:50:18.890 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:50:18.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-10-30T11:50:18.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:50:18.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:50:19.318 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:50:19.319 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:50:19.319 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:50:19.319 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-5.14.0-378.el9.x86_64 2023-10-30T11:50:19.319 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-10-30T11:50:19.320 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-10-30T11:50:19.320 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-10-30T11:50:19.320 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:50:19.320 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:50:19.801 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T11:50:19.942 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T11:50:19.942 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T11:50:19.942 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-10-30T11:50:19.943 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-30T11:50:19.943 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-30T11:50:19.943 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-10-30T11:50:19.944 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi092.front.sepia.ceph.com, path=None, version=distro) 2023-10-30T11:50:19.944 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T11:50:20.060 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T11:50:20.060 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T11:50:20.061 DEBUG:teuthology.orchestra.run.smithi092:> sudo rpm -qi grub2-tools 2023-10-30T11:50:20.103 INFO:teuthology.orchestra.run.smithi092.stdout:Name : grub2-tools 2023-10-30T11:50:20.104 INFO:teuthology.orchestra.run.smithi092.stdout:Epoch : 1 2023-10-30T11:50:20.104 INFO:teuthology.orchestra.run.smithi092.stdout:Version : 2.06 2023-10-30T11:50:20.104 INFO:teuthology.orchestra.run.smithi092.stdout:Release : 68.el9 2023-10-30T11:50:20.104 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: x86_64 2023-10-30T11:50:20.105 INFO:teuthology.orchestra.run.smithi092.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-30T11:50:20.105 INFO:teuthology.orchestra.run.smithi092.stdout:Group : Unspecified 2023-10-30T11:50:20.105 INFO:teuthology.orchestra.run.smithi092.stdout:Size : 8264254 2023-10-30T11:50:20.105 INFO:teuthology.orchestra.run.smithi092.stdout:License : GPLv3+ 2023-10-30T11:50:20.105 INFO:teuthology.orchestra.run.smithi092.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-30T11:50:20.106 INFO:teuthology.orchestra.run.smithi092.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-30T11:50:20.106 INFO:teuthology.orchestra.run.smithi092.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-30T11:50:20.106 INFO:teuthology.orchestra.run.smithi092.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-30T11:50:20.106 INFO:teuthology.orchestra.run.smithi092.stdout:Packager : builder@centos.org 2023-10-30T11:50:20.106 INFO:teuthology.orchestra.run.smithi092.stdout:Vendor : CentOS 2023-10-30T11:50:20.107 INFO:teuthology.orchestra.run.smithi092.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-30T11:50:20.107 INFO:teuthology.orchestra.run.smithi092.stdout:Summary : Support tools for GRUB. 2023-10-30T11:50:20.107 INFO:teuthology.orchestra.run.smithi092.stdout:Description : 2023-10-30T11:50:20.107 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:50:20.107 INFO:teuthology.orchestra.run.smithi092.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-30T11:50:20.108 INFO:teuthology.orchestra.run.smithi092.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-30T11:50:20.108 INFO:teuthology.orchestra.run.smithi092.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-30T11:50:20.108 INFO:teuthology.orchestra.run.smithi092.stdout:hardware devices. 2023-10-30T11:50:20.108 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:50:20.108 INFO:teuthology.orchestra.run.smithi092.stdout:This subpackage provides tools for support of all platforms. 2023-10-30T11:50:20.109 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-30T11:50:20.110 INFO:teuthology.task.kernel:Updating grub on smithi092 to boot 5.14.0-378.el9.x86_64 2023-10-30T11:50:20.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-30T11:50:20.693 INFO:teuthology.orchestra.run.smithi092.stderr:Generating grub configuration file ... 2023-10-30T11:50:22.065 INFO:teuthology.orchestra.run.smithi092.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-30T11:50:22.104 INFO:teuthology.orchestra.run.smithi092.stderr:done 2023-10-30T11:50:22.117 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2023-10-30T11:50:22.133 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.ZuG71TpAWG 2023-10-30T11:50:22.134 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZuG71TpAWG 2023-10-30T11:50:22.202 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.ZuG71TpAWG 2023-10-30T11:50:22.341 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.ZuG71TpAWG is 6KB 2023-10-30T11:50:22.360 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.ZuG71TpAWG 2023-10-30T11:50:22.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo /bin/ls /boot/loader/entries || true 2023-10-30T11:50:22.448 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-30T11:50:22.449 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-30T11:50:22.449 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-30T11:50:22.449 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-10-30T11:50:22.450 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-10-30T11:50:22.585 DEBUG:teuthology.orchestra.run.smithi092:> sudo shutdown -r now 2023-10-30T11:50:26.747 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:50:26.791 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:50:27.217 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:50:27.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-10-30T11:50:27.218 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:50:27.218 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:50:27.624 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:50:27.624 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:50:27.624 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:50:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-5.14.0-378.el9.x86_64 2023-10-30T11:50:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-10-30T11:50:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-10-30T11:50:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-10-30T11:50:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:50:27.626 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:50:28.133 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T11:50:28.274 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T11:50:28.275 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T11:50:28.275 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-10-30T11:50:28.275 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-30T11:50:28.275 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-30T11:50:28.276 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-30T11:50:28.276 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2023-10-30T11:50:28.276 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T11:50:28.398 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T11:50:28.398 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T11:50:28.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -qi grub2-tools 2023-10-30T11:50:28.436 INFO:teuthology.orchestra.run.smithi003.stdout:Name : grub2-tools 2023-10-30T11:50:28.437 INFO:teuthology.orchestra.run.smithi003.stdout:Epoch : 1 2023-10-30T11:50:28.437 INFO:teuthology.orchestra.run.smithi003.stdout:Version : 2.06 2023-10-30T11:50:28.437 INFO:teuthology.orchestra.run.smithi003.stdout:Release : 68.el9 2023-10-30T11:50:28.437 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: x86_64 2023-10-30T11:50:28.437 INFO:teuthology.orchestra.run.smithi003.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-30T11:50:28.438 INFO:teuthology.orchestra.run.smithi003.stdout:Group : Unspecified 2023-10-30T11:50:28.438 INFO:teuthology.orchestra.run.smithi003.stdout:Size : 8264254 2023-10-30T11:50:28.438 INFO:teuthology.orchestra.run.smithi003.stdout:License : GPLv3+ 2023-10-30T11:50:28.438 INFO:teuthology.orchestra.run.smithi003.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-30T11:50:28.438 INFO:teuthology.orchestra.run.smithi003.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-30T11:50:28.439 INFO:teuthology.orchestra.run.smithi003.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-30T11:50:28.439 INFO:teuthology.orchestra.run.smithi003.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-30T11:50:28.439 INFO:teuthology.orchestra.run.smithi003.stdout:Packager : builder@centos.org 2023-10-30T11:50:28.439 INFO:teuthology.orchestra.run.smithi003.stdout:Vendor : CentOS 2023-10-30T11:50:28.439 INFO:teuthology.orchestra.run.smithi003.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-30T11:50:28.440 INFO:teuthology.orchestra.run.smithi003.stdout:Summary : Support tools for GRUB. 2023-10-30T11:50:28.440 INFO:teuthology.orchestra.run.smithi003.stdout:Description : 2023-10-30T11:50:28.440 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:50:28.440 INFO:teuthology.orchestra.run.smithi003.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-30T11:50:28.440 INFO:teuthology.orchestra.run.smithi003.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-30T11:50:28.441 INFO:teuthology.orchestra.run.smithi003.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-30T11:50:28.441 INFO:teuthology.orchestra.run.smithi003.stdout:hardware devices. 2023-10-30T11:50:28.441 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:50:28.441 INFO:teuthology.orchestra.run.smithi003.stdout:This subpackage provides tools for support of all platforms. 2023-10-30T11:50:28.442 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-30T11:50:28.445 INFO:teuthology.task.kernel:Updating grub on smithi003 to boot 5.14.0-378.el9.x86_64 2023-10-30T11:50:28.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-30T11:50:29.079 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2023-10-30T11:50:30.462 INFO:teuthology.orchestra.run.smithi003.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-30T11:50:30.493 INFO:teuthology.orchestra.run.smithi003.stderr:done 2023-10-30T11:50:30.495 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2023-10-30T11:50:30.511 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.lM2UmZOlCZ 2023-10-30T11:50:30.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lM2UmZOlCZ 2023-10-30T11:50:30.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.lM2UmZOlCZ 2023-10-30T11:50:30.735 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.lM2UmZOlCZ is 6KB 2023-10-30T11:50:30.784 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.lM2UmZOlCZ 2023-10-30T11:50:30.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo /bin/ls /boot/loader/entries || true 2023-10-30T11:50:30.873 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-30T11:50:30.874 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-30T11:50:30.874 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-30T11:50:30.874 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-10-30T11:50:30.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-10-30T11:50:31.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2023-10-30T11:50:52.606 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-10-30T11:50:52.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-10-30T11:50:52.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:51:01.050 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-30T11:51:01.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-10-30T11:51:01.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:51:52.664 DEBUG:teuthology.orchestra.remote:timed out 2023-10-30T11:52:01.058 DEBUG:teuthology.orchestra.remote:timed out 2023-10-30T11:52:01.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-10-30T11:52:01.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:52:05.232 DEBUG:teuthology.orchestra.run.smithi092:> true 2023-10-30T11:52:05.651 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-10-30T11:52:05.652 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-378.el9.x86_64"... 2023-10-30T11:52:05.652 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2023-10-30T11:52:05.666 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-378.el9.x86_64 2023-10-30T11:52:05.666 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-10-30T11:52:05.666 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T11:52:05.667 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-30T11:52:06.668 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-10-30T11:52:06.669 DEBUG:teuthology.orchestra.run.smithi092:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-30T11:52:06.731 INFO:teuthology.orchestra.run.smithi092.stdout:ttyS1 2023-10-30T11:52:06.760 DEBUG:teuthology.parallel:result is None 2023-10-30T11:52:10.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-10-30T11:52:10.060 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:52:10.390 DEBUG:teuthology.orchestra.run.smithi003:> true 2023-10-30T11:52:10.806 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-10-30T11:52:10.806 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-378.el9.x86_64"... 2023-10-30T11:52:10.806 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2023-10-30T11:52:10.821 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-378.el9.x86_64 2023-10-30T11:52:10.822 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-10-30T11:52:10.822 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T11:52:10.822 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-30T11:52:11.823 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-30T11:52:11.824 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-30T11:52:11.877 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2023-10-30T11:52:11.928 DEBUG:teuthology.parallel:result is None 2023-10-30T11:52:11.929 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-30T11:52:11.936 INFO:teuthology.task.internal:Creating test directory... 2023-10-30T11:52:11.936 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T11:52:11.939 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T11:52:11.955 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-30T11:52:11.962 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-30T11:52:11.971 INFO:teuthology.task.internal:Creating archive directory... 2023-10-30T11:52:11.971 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T11:52:11.993 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T11:52:12.027 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-30T11:52:12.037 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-30T11:52:12.037 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T11:52:12.054 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T11:52:12.091 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T11:52:12.092 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T11:52:12.104 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T11:52:12.109 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T11:52:12.110 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-30T11:52:12.118 INFO:teuthology.task.internal:Configuring sudo... 2023-10-30T11:52:12.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T11:52:12.152 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T11:52:12.180 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-30T11:52:12.190 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-30T11:52:12.190 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T11:52:12.223 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T11:52:12.238 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T11:52:12.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T11:52:12.370 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T11:52:12.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T11:52:12.498 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T11:52:12.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T11:52:12.571 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T11:52:12.607 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T11:52:12.662 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T11:52:12.743 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T11:52:12.784 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T11:52:12.784 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T11:52:12.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2023-10-30T11:52:12.855 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2023-10-30T11:52:12.905 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T11:52:12.949 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T11:52:13.444 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-30T11:52:13.452 INFO:teuthology.task.internal:Starting timer... 2023-10-30T11:52:13.452 INFO:teuthology.run_tasks:Running task pcp... 2023-10-30T11:52:13.461 INFO:teuthology.run_tasks:Running task selinux... 2023-10-30T11:52:13.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2023-10-30T11:52:13.544 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2023-10-30T11:52:13.545 DEBUG:teuthology.orchestra.run.smithi092:> sudo service auditd rotate 2023-10-30T11:52:13.613 INFO:teuthology.orchestra.run.smithi092.stdout:Rotating logs: 2023-10-30T11:52:13.615 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-30T11:52:13.615 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2023-10-30T11:52:13.634 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2023-10-30T11:52:13.634 DEBUG:teuthology.orchestra.run.smithi092:> /usr/sbin/getenforce 2023-10-30T11:52:13.655 INFO:teuthology.orchestra.run.smithi092.stdout:Permissive 2023-10-30T11:52:13.656 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi092.front.sepia.ceph.com': 'permissive'} 2023-10-30T11:52:13.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T11:52:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T11:52:13.707 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T11:52:13.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T11:52:13.736 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-30T11:52:13.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2023-10-30T11:52:13.786 DEBUG:teuthology.orchestra.run.smithi092:> sudo /usr/sbin/setenforce permissive 2023-10-30T11:52:13.816 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-30T11:52:13.825 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-30T11:52:13.826 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-30T11:52:13.842 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-30T11:52:13.843 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi092.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-30T11:57:46.764 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi092.front.sepia.ceph.com')] 2023-10-30T11:57:46.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-10-30T11:57:46.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:57:46.865 DEBUG:teuthology.orchestra.run.smithi003:> true 2023-10-30T11:57:46.933 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-10-30T11:57:46.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-10-30T11:57:46.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:57:47.018 DEBUG:teuthology.orchestra.run.smithi092:> true 2023-10-30T11:57:47.089 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-10-30T11:57:47.089 INFO:teuthology.run_tasks:Running task clock... 2023-10-30T11:57:47.103 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-30T11:57:47.103 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T11:57:47.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T11:57:47.107 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T11:57:47.107 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T11:57:47.139 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-30T11:57:47.161 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-30T11:57:47.162 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-30T11:57:47.177 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-30T11:57:47.196 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2023-10-30T11:57:47.206 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: ntpd: command not found 2023-10-30T11:57:47.213 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2023-10-30T11:57:47.219 INFO:teuthology.orchestra.run.smithi092.stdout:506 Cannot talk to daemon 2023-10-30T11:57:47.234 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-30T11:57:47.235 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-30T11:57:47.251 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-30T11:57:47.253 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-30T11:57:47.294 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2023-10-30T11:57:47.297 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T11:57:47.297 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2023-10-30T11:57:47.297 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:47.298 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:47.298 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:47.298 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:47.301 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2023-10-30T11:57:47.305 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T11:57:47.305 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2023-10-30T11:57:47.305 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:47.306 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:47.306 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:47.306 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:47.306 INFO:teuthology.run_tasks:Running task install... 2023-10-30T11:57:47.314 DEBUG:teuthology.task.install:project ceph 2023-10-30T11:57:47.315 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d'}} 2023-10-30T11:57:47.315 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d'} 2023-10-30T11:57:47.315 INFO:teuthology.task.install:Using flavor: default 2023-10-30T11:57:47.321 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-10-30T11:57:47.321 INFO:teuthology.task.install:extra packages: [] 2023-10-30T11:57:47.322 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d', 'tag': None, 'wait_for_package': False} 2023-10-30T11:57:47.323 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cc36dd85d142c6fdb77515cad8af108cec98a86d 2023-10-30T11:57:47.325 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d', 'tag': None, 'wait_for_package': False} 2023-10-30T11:57:47.325 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cc36dd85d142c6fdb77515cad8af108cec98a86d 2023-10-30T11:57:47.546 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/ 2023-10-30T11:57:47.547 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6839.gcc36dd85 2023-10-30T11:57:47.555 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/ 2023-10-30T11:57:47.555 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6839.gcc36dd85 2023-10-30T11:57:47.658 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-30T11:57:47.659 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T11:57:47.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-30T11:57:47.692 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-10-30T11:57:47.693 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cc36dd85d142c6fdb77515cad8af108cec98a86d/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-30T11:57:47.729 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/cc36dd85d142c6fdb77515cad8af108cec98a86d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-30T11:57:47.729 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T11:57:47.730 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-30T11:57:47.758 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-10-30T11:57:47.758 DEBUG:teuthology.orchestra.run.smithi092:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cc36dd85d142c6fdb77515cad8af108cec98a86d/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-30T11:57:47.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-30T11:57:47.826 DEBUG:teuthology.orchestra.run.smithi092:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-30T11:57:47.851 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-30T11:57:47.881 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2023-10-30T11:57:47.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2023-10-30T11:57:47.905 DEBUG:teuthology.orchestra.run.smithi092:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-30T11:57:47.971 INFO:teuthology.orchestra.run.smithi092.stdout:check_obsoletes = 1 2023-10-30T11:57:47.972 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum clean all 2023-10-30T11:57:48.128 INFO:teuthology.orchestra.run.smithi003.stdout:72 files removed 2023-10-30T11:57:48.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw 2023-10-30T11:57:48.210 INFO:teuthology.orchestra.run.smithi092.stdout:72 files removed 2023-10-30T11:57:48.236 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-radosgw 2023-10-30T11:57:48.859 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 212 kB/s | 78 kB 00:00 2023-10-30T11:57:48.941 INFO:teuthology.orchestra.run.smithi092.stdout:ceph packages for x86_64 200 kB/s | 78 kB 00:00 2023-10-30T11:57:49.251 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 28 kB/s | 9.4 kB 00:00 2023-10-30T11:57:49.275 INFO:teuthology.orchestra.run.smithi092.stdout:ceph noarch packages 33 kB/s | 9.4 kB 00:00 2023-10-30T11:57:49.627 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2023-10-30T11:57:49.684 INFO:teuthology.orchestra.run.smithi092.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2023-10-30T11:57:50.021 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 18 MB/s | 6.3 MB 00:00 2023-10-30T11:57:50.146 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - AppStream 15 MB/s | 6.3 MB 00:00 2023-10-30T11:57:51.379 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2023-10-30T11:57:51.454 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2023-10-30T11:57:52.246 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 7.9 MB 00:00 2023-10-30T11:57:54.586 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 7.9 MB 00:02 2023-10-30T11:57:55.438 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 8.5 MB/s | 18 MB 00:02 2023-10-30T11:57:57.979 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 7.5 MB/s | 18 MB 00:02 2023-10-30T11:58:00.864 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 5.7 MB 00:02 2023-10-30T11:58:01.887 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 5.7 MB 00:00 2023-10-30T11:58:02.633 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 15 kB 00:00 2023-10-30T11:58:04.421 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 15 kB 00:01 2023-10-30T11:58:05.650 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 17 MB/s | 19 MB 00:01 2023-10-30T11:58:06.727 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 4.9 MB/s | 19 MB 00:03 2023-10-30T11:58:10.181 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2023-10-30T11:58:11.408 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2023-10-30T11:58:12.550 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:58:12.559 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:58:12.559 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-10-30T11:58:12.559 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:58:12.560 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:58:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 8.7 M 2023-10-30T11:58:12.560 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrading: 2023-10-30T11:58:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: librados2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.3 M 2023-10-30T11:58:12.561 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.0 M 2023-10-30T11:58:12.561 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-10-30T11:58:12.561 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options 2023-10-30T11:58:12.561 INFO:teuthology.orchestra.run.smithi092.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-30T11:58:12.561 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.2 M 2023-10-30T11:58:12.562 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 19 M 2023-10-30T11:58:12.562 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 24 k 2023-10-30T11:58:12.562 INFO:teuthology.orchestra.run.smithi092.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-30T11:58:12.562 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-30T11:58:12.563 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-30T11:58:12.563 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-30T11:58:12.563 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 715 k 2023-10-30T11:58:12.563 INFO:teuthology.orchestra.run.smithi092.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-30T11:58:12.563 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-30T11:58:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-30T11:58:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 472 k 2023-10-30T11:58:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-30T11:58:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 4.8 M 2023-10-30T11:58:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-30T11:58:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-30T11:58:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-30T11:58:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse 2023-10-30T11:58:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 45 k 2023-10-30T11:58:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 127 k 2023-10-30T11:58:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 162 k 2023-10-30T11:58:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-30T11:58:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 321 k 2023-10-30T11:58:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 298 k 2023-10-30T11:58:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 99 k 2023-10-30T11:58:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-30T11:58:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-30T11:58:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:58:12.567 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:58:12.568 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:58:12.568 INFO:teuthology.orchestra.run.smithi092.stdout:Install 28 Packages 2023-10-30T11:58:12.568 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrade 2 Packages 2023-10-30T11:58:12.568 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:58:12.569 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 55 M 2023-10-30T11:58:12.569 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:58:13.885 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:58:13.895 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:58:13.895 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-30T11:58:13.895 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:58:13.895 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:58:13.896 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 8.7 M 2023-10-30T11:58:13.896 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2023-10-30T11:58:13.896 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.3 M 2023-10-30T11:58:13.896 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.0 M 2023-10-30T11:58:13.896 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-30T11:58:13.897 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options 2023-10-30T11:58:13.898 INFO:teuthology.orchestra.run.smithi003.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-30T11:58:13.898 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.2 M 2023-10-30T11:58:13.898 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 19 M 2023-10-30T11:58:13.898 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 24 k 2023-10-30T11:58:13.899 INFO:teuthology.orchestra.run.smithi003.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-30T11:58:13.899 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-30T11:58:13.899 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-30T11:58:13.899 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-30T11:58:13.899 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 715 k 2023-10-30T11:58:13.900 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-30T11:58:13.900 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-30T11:58:13.900 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-30T11:58:13.900 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 472 k 2023-10-30T11:58:13.900 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-30T11:58:13.901 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 4.8 M 2023-10-30T11:58:13.901 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-30T11:58:13.901 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-30T11:58:13.901 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-30T11:58:13.901 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse 2023-10-30T11:58:13.902 INFO:teuthology.orchestra.run.smithi003.stdout: x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 45 k 2023-10-30T11:58:13.902 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 127 k 2023-10-30T11:58:13.902 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 162 k 2023-10-30T11:58:13.902 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-30T11:58:13.902 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 321 k 2023-10-30T11:58:13.903 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 298 k 2023-10-30T11:58:13.903 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 99 k 2023-10-30T11:58:13.903 INFO:teuthology.orchestra.run.smithi003.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-30T11:58:13.903 INFO:teuthology.orchestra.run.smithi003.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-30T11:58:13.903 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:58:13.904 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:58:13.904 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:58:13.904 INFO:teuthology.orchestra.run.smithi003.stdout:Install 28 Packages 2023-10-30T11:58:13.904 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 2 Packages 2023-10-30T11:58:13.904 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:58:13.905 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 55 M 2023-10-30T11:58:13.905 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:58:14.406 INFO:teuthology.orchestra.run.smithi092.stdout:(1/30): ceph-base-18.0.0-6839.gcc36dd85.el9.x86 3.8 MB/s | 5.2 MB 00:01 2023-10-30T11:58:14.465 INFO:teuthology.orchestra.run.smithi092.stdout:(2/30): ceph-selinux-18.0.0-6839.gcc36dd85.el9. 412 kB/s | 24 kB 00:00 2023-10-30T11:58:14.615 INFO:teuthology.orchestra.run.smithi092.stdout:(3/30): libcephfs2-18.0.0-6839.gcc36dd85.el9.x8 4.7 MB/s | 715 kB 00:00 2023-10-30T11:58:14.708 INFO:teuthology.orchestra.run.smithi092.stdout:(4/30): libradosstriper1-18.0.0-6839.gcc36dd85. 5.0 MB/s | 472 kB 00:00 2023-10-30T11:58:15.391 INFO:teuthology.orchestra.run.smithi092.stdout:(5/30): ceph-radosgw-18.0.0-6839.gcc36dd85.el9. 3.7 MB/s | 8.7 MB 00:02 2023-10-30T11:58:15.458 INFO:teuthology.orchestra.run.smithi092.stdout:(6/30): librgw2-18.0.0-6839.gcc36dd85.el9.x86_6 6.4 MB/s | 4.8 MB 00:00 2023-10-30T11:58:15.475 INFO:teuthology.orchestra.run.smithi092.stdout:(7/30): python3-ceph-argparse-18.0.0-6839.gcc36 542 kB/s | 45 kB 00:00 2023-10-30T11:58:15.517 INFO:teuthology.orchestra.run.smithi092.stdout:(8/30): python3-ceph-common-18.0.0-6839.gcc36dd 2.1 MB/s | 127 kB 00:00 2023-10-30T11:58:15.542 INFO:teuthology.orchestra.run.smithi092.stdout:(9/30): python3-cephfs-18.0.0-6839.gcc36dd85.el 2.4 MB/s | 162 kB 00:00 2023-10-30T11:58:15.584 INFO:teuthology.orchestra.run.smithi092.stdout:(10/30): python3-rados-18.0.0-6839.gcc36dd85.el 4.7 MB/s | 321 kB 00:00 2023-10-30T11:58:15.651 INFO:teuthology.orchestra.run.smithi092.stdout:(11/30): python3-rgw-18.0.0-6839.gcc36dd85.el9. 1.5 MB/s | 99 kB 00:00 2023-10-30T11:58:15.676 INFO:teuthology.orchestra.run.smithi092.stdout:(12/30): python3-rbd-18.0.0-6839.gcc36dd85.el9. 2.2 MB/s | 298 kB 00:00 2023-10-30T11:58:15.785 INFO:teuthology.orchestra.run.smithi092.stdout:(13/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 159 kB 00:00 2023-10-30T11:58:15.810 INFO:teuthology.orchestra.run.smithi092.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 666 kB/s | 106 kB 00:00 2023-10-30T11:58:15.844 INFO:teuthology.orchestra.run.smithi092.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-10-30T11:58:15.869 INFO:teuthology.orchestra.run.smithi092.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2023-10-30T11:58:15.875 INFO:teuthology.orchestra.run.smithi003.stdout:(1/30): ceph-base-18.0.0-6839.gcc36dd85.el9.x86 3.8 MB/s | 5.2 MB 00:01 2023-10-30T11:58:15.894 INFO:teuthology.orchestra.run.smithi092.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 824 kB/s | 41 kB 00:00 2023-10-30T11:58:15.934 INFO:teuthology.orchestra.run.smithi003.stdout:(2/30): ceph-selinux-18.0.0-6839.gcc36dd85.el9. 414 kB/s | 24 kB 00:00 2023-10-30T11:58:16.051 INFO:teuthology.orchestra.run.smithi003.stdout:(3/30): libcephfs2-18.0.0-6839.gcc36dd85.el9.x8 6.0 MB/s | 715 kB 00:00 2023-10-30T11:58:16.061 INFO:teuthology.orchestra.run.smithi092.stdout:(18/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 236 kB/s | 45 kB 00:00 2023-10-30T11:58:16.152 INFO:teuthology.orchestra.run.smithi003.stdout:(4/30): libradosstriper1-18.0.0-6839.gcc36dd85. 4.6 MB/s | 472 kB 00:00 2023-10-30T11:58:16.603 INFO:teuthology.orchestra.run.smithi092.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 156 kB/s | 111 kB 00:00 2023-10-30T11:58:16.837 INFO:teuthology.orchestra.run.smithi092.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 398 kB/s | 309 kB 00:00 2023-10-30T11:58:16.909 INFO:teuthology.orchestra.run.smithi003.stdout:(5/30): librgw2-18.0.0-6839.gcc36dd85.el9.x86_6 6.3 MB/s | 4.8 MB 00:00 2023-10-30T11:58:16.968 INFO:teuthology.orchestra.run.smithi003.stdout:(6/30): python3-ceph-argparse-18.0.0-6839.gcc36 777 kB/s | 45 kB 00:00 2023-10-30T11:58:17.193 INFO:teuthology.orchestra.run.smithi003.stdout:(7/30): ceph-common-18.0.0-6839.gcc36dd85.el9.x 7.1 MB/s | 19 MB 00:02 2023-10-30T11:58:17.210 INFO:teuthology.orchestra.run.smithi003.stdout:(8/30): python3-ceph-common-18.0.0-6839.gcc36dd 523 kB/s | 127 kB 00:00 2023-10-30T11:58:17.244 INFO:teuthology.orchestra.run.smithi003.stdout:(9/30): python3-cephfs-18.0.0-6839.gcc36dd85.el 3.2 MB/s | 162 kB 00:00 2023-10-30T11:58:17.269 INFO:teuthology.orchestra.run.smithi003.stdout:(10/30): python3-rados-18.0.0-6839.gcc36dd85.el 5.4 MB/s | 321 kB 00:00 2023-10-30T11:58:17.294 INFO:teuthology.orchestra.run.smithi003.stdout:(11/30): python3-rbd-18.0.0-6839.gcc36dd85.el9. 5.8 MB/s | 298 kB 00:00 2023-10-30T11:58:17.312 INFO:teuthology.orchestra.run.smithi092.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 51 kB/s | 24 kB 00:00 2023-10-30T11:58:17.319 INFO:teuthology.orchestra.run.smithi003.stdout:(12/30): python3-rgw-18.0.0-6839.gcc36dd85.el9. 1.9 MB/s | 99 kB 00:00 2023-10-30T11:58:17.420 INFO:teuthology.orchestra.run.smithi003.stdout:(13/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.5 MB/s | 159 kB 00:00 2023-10-30T11:58:17.445 INFO:teuthology.orchestra.run.smithi003.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 702 kB/s | 106 kB 00:00 2023-10-30T11:58:17.487 INFO:teuthology.orchestra.run.smithi003.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2023-10-30T11:58:17.513 INFO:teuthology.orchestra.run.smithi003.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2023-10-30T11:58:17.538 INFO:teuthology.orchestra.run.smithi003.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2023-10-30T11:58:17.738 INFO:teuthology.orchestra.run.smithi003.stdout:(18/30): fmt-8.1.1-5.el9.x86_64.rpm 554 kB/s | 111 kB 00:00 2023-10-30T11:58:17.764 INFO:teuthology.orchestra.run.smithi003.stdout:(19/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 181 kB/s | 45 kB 00:00 2023-10-30T11:58:17.854 INFO:teuthology.orchestra.run.smithi092.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 90 kB/s | 49 kB 00:00 2023-10-30T11:58:17.863 INFO:teuthology.orchestra.run.smithi003.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.4 MB/s | 309 kB 00:00 2023-10-30T11:58:17.922 INFO:teuthology.orchestra.run.smithi003.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 417 kB/s | 24 kB 00:00 2023-10-30T11:58:17.981 INFO:teuthology.orchestra.run.smithi003.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 831 kB/s | 49 kB 00:00 2023-10-30T11:58:18.056 INFO:teuthology.orchestra.run.smithi003.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 900 kB/s | 67 kB 00:00 2023-10-30T11:58:18.196 INFO:teuthology.orchestra.run.smithi092.stdout:(23/30): ceph-common-18.0.0-6839.gcc36dd85.el9. 3.7 MB/s | 19 MB 00:05 2023-10-30T11:58:18.231 INFO:teuthology.orchestra.run.smithi003.stdout:(24/30): ceph-radosgw-18.0.0-6839.gcc36dd85.el9 2.3 MB/s | 8.7 MB 00:03 2023-10-30T11:58:18.265 INFO:teuthology.orchestra.run.smithi003.stdout:(25/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.9 MB/s | 837 kB 00:00 2023-10-30T11:58:18.340 INFO:teuthology.orchestra.run.smithi003.stdout:(26/30): libarrow-9.0.0-9.el9.x86_64.rpm 7.7 MB/s | 4.4 MB 00:00 2023-10-30T11:58:18.449 INFO:teuthology.orchestra.run.smithi003.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2023-10-30T11:58:18.571 INFO:teuthology.orchestra.run.smithi092.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 94 kB/s | 67 kB 00:00 2023-10-30T11:58:18.649 INFO:teuthology.orchestra.run.smithi003.stdout:(28/30): librados2-18.0.0-6839.gcc36dd85.el9.x8 11 MB/s | 3.3 MB 00:00 2023-10-30T11:58:18.674 INFO:teuthology.orchestra.run.smithi003.stdout:(29/30): re2-20211101-3.el9.x86_64.rpm 433 kB/s | 192 kB 00:00 2023-10-30T11:58:18.758 INFO:teuthology.orchestra.run.smithi003.stdout:(30/30): librbd1-18.0.0-6839.gcc36dd85.el9.x86_ 9.8 MB/s | 3.0 MB 00:00 2023-10-30T11:58:18.761 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:58:18.762 INFO:teuthology.orchestra.run.smithi003.stdout:Total 11 MB/s | 55 MB 00:04 2023-10-30T11:58:18.775 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:58:18.816 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:58:18.817 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:58:19.307 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:58:19.308 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:58:19.430 INFO:teuthology.orchestra.run.smithi092.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 223 kB/s | 192 kB 00:00 2023-10-30T11:58:19.655 INFO:teuthology.orchestra.run.smithi092.stdout:(26/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 575 kB/s | 837 kB 00:01 2023-10-30T11:58:20.013 INFO:teuthology.orchestra.run.smithi092.stdout:(27/30): librados2-18.0.0-6839.gcc36dd85.el9.x8 9.2 MB/s | 3.3 MB 00:00 2023-10-30T11:58:20.255 INFO:teuthology.orchestra.run.smithi092.stdout:(28/30): librbd1-18.0.0-6839.gcc36dd85.el9.x86_ 13 MB/s | 3.0 MB 00:00 2023-10-30T11:58:20.480 INFO:teuthology.orchestra.run.smithi092.stdout:(29/30): libarrow-9.0.0-9.el9.x86_64.rpm 1.1 MB/s | 4.4 MB 00:03 2023-10-30T11:58:20.819 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:58:20.871 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-30T11:58:20.928 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-10-30T11:58:20.930 INFO:teuthology.orchestra.run.smithi092.stdout:(30/30): thrift-0.15.0-2.el9.x86_64.rpm 1.1 MB/s | 1.6 MB 00:01 2023-10-30T11:58:20.935 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:58:20.935 INFO:teuthology.orchestra.run.smithi092.stdout:Total 6.6 MB/s | 55 MB 00:08 2023-10-30T11:58:20.947 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:58:20.981 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-10-30T11:58:20.985 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:58:20.985 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:58:21.374 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-10-30T11:58:21.404 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T11:58:21.456 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:58:21.457 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:58:21.496 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T11:58:21.583 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/32 2023-10-30T11:58:21.614 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T11:58:21.676 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T11:58:21.705 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-10-30T11:58:21.749 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9. 9/32 2023-10-30T11:58:21.816 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 10/32 2023-10-30T11:58:21.841 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 11/32 2023-10-30T11:58:21.907 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 11/32 2023-10-30T11:58:22.016 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-30T11:58:22.049 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-30T11:58:22.089 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-30T11:58:22.121 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-30T11:58:22.148 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-30T11:58:22.184 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-30T11:58:22.222 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T11:58:22.541 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-30T11:58:22.569 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 20/32 2023-10-30T11:58:22.660 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 20/32 2023-10-30T11:58:22.690 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 21/32 2023-10-30T11:58:22.727 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-30T11:58:22.846 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:58:22.910 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-30T11:58:22.952 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-10-30T11:58:23.011 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-10-30T11:58:23.178 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-30T11:58:23.205 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 24/32 2023-10-30T11:58:23.259 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 24/32 2023-10-30T11:58:23.327 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-10-30T11:58:23.330 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 25/32 2023-10-30T11:58:23.357 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x8 26/32 2023-10-30T11:58:23.376 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T11:58:23.458 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T11:58:23.544 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/32 2023-10-30T11:58:23.569 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T11:58:23.618 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T11:58:23.652 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-10-30T11:58:23.696 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9. 9/32 2023-10-30T11:58:23.771 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 10/32 2023-10-30T11:58:23.797 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 11/32 2023-10-30T11:58:23.862 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 11/32 2023-10-30T11:58:23.976 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-30T11:58:24.005 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-30T11:58:24.045 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-30T11:58:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-30T11:58:24.104 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-30T11:58:24.132 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-30T11:58:24.165 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T11:58:24.475 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-30T11:58:24.503 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 20/32 2023-10-30T11:58:24.612 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 20/32 2023-10-30T11:58:24.643 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 21/32 2023-10-30T11:58:24.671 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-30T11:58:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-30T11:58:25.124 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 24/32 2023-10-30T11:58:25.195 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 24/32 2023-10-30T11:58:25.263 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 25/32 2023-10-30T11:58:25.294 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x8 26/32 2023-10-30T11:58:25.584 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T11:58:25.621 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T11:58:26.204 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T11:58:26.328 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 28/32 2023-10-30T11:58:26.416 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 28/32 2023-10-30T11:58:26.417 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-30T11:58:26.417 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-30T11:58:26.417 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:58:26.454 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T11:58:27.384 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T11:58:27.424 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T11:58:27.971 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 27/32 2023-10-30T11:58:28.011 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 28/32 2023-10-30T11:58:28.095 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 28/32 2023-10-30T11:58:28.095 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-30T11:58:28.095 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-30T11:58:28.096 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:58:28.128 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T11:58:36.523 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T11:58:36.524 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2023-10-30T11:58:36.524 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2023-10-30T11:58:36.524 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2023-10-30T11:58:36.524 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2023-10-30T11:58:36.524 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2023-10-30T11:58:36.525 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /root 2023-10-30T11:58:36.525 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2023-10-30T11:58:36.525 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:58:37.369 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/32 2023-10-30T11:58:37.395 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/32 2023-10-30T11:58:37.395 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:58:37.396 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-30T11:58:37.396 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T11:58:37.396 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T11:58:37.396 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:58:37.397 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T11:58:37.460 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T11:58:37.460 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /sys 2023-10-30T11:58:37.460 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /proc 2023-10-30T11:58:37.461 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /mnt 2023-10-30T11:58:37.461 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /var/tmp 2023-10-30T11:58:37.461 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /home 2023-10-30T11:58:37.461 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /root 2023-10-30T11:58:37.461 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /tmp 2023-10-30T11:58:37.462 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:58:37.480 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T11:58:37.480 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T11:58:38.290 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/32 2023-10-30T11:58:38.314 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/32 2023-10-30T11:58:38.315 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:58:38.315 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-30T11:58:38.315 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T11:58:38.316 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T11:58:38.316 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:58:38.316 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T11:58:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T11:58:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T11:58:38.565 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T11:58:38.565 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/32 2023-10-30T11:58:38.565 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/32 2023-10-30T11:58:38.566 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 3/32 2023-10-30T11:58:38.566 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 4/32 2023-10-30T11:58:38.566 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T11:58:38.566 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 6/32 2023-10-30T11:58:38.566 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T11:58:38.567 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9. 8/32 2023-10-30T11:58:38.567 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x8 9/32 2023-10-30T11:58:38.567 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 10/32 2023-10-30T11:58:38.567 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 11/32 2023-10-30T11:58:38.567 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 12/32 2023-10-30T11:58:38.568 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 13/32 2023-10-30T11:58:38.568 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-30T11:58:38.568 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-30T11:58:38.568 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-30T11:58:38.568 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-30T11:58:38.569 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T11:58:38.569 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-30T11:58:38.569 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-30T11:58:38.569 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-30T11:58:38.569 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-30T11:58:38.570 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-30T11:58:38.570 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-30T11:58:38.570 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-30T11:58:38.570 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-30T11:58:38.570 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-30T11:58:38.571 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-30T11:58:38.571 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T11:58:38.571 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-30T11:58:38.572 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/32 2023-10-30T11:58:39.505 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T11:58:39.505 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/32 2023-10-30T11:58:39.505 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/32 2023-10-30T11:58:39.506 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 3/32 2023-10-30T11:58:39.506 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 4/32 2023-10-30T11:58:39.506 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/32 2023-10-30T11:58:39.506 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_6 6/32 2023-10-30T11:58:39.507 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 7/32 2023-10-30T11:58:39.507 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9. 8/32 2023-10-30T11:58:39.507 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x8 9/32 2023-10-30T11:58:39.507 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 10/32 2023-10-30T11:58:39.507 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 11/32 2023-10-30T11:58:39.508 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 12/32 2023-10-30T11:58:39.508 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 13/32 2023-10-30T11:58:39.508 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-30T11:58:39.508 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-30T11:58:39.508 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-30T11:58:39.509 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-30T11:58:39.509 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T11:58:39.509 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-30T11:58:39.509 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-30T11:58:39.509 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-30T11:58:39.509 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-30T11:58:39.510 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-30T11:58:39.510 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-30T11:58:39.510 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-30T11:58:39.510 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-30T11:58:39.510 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-30T11:58:39.511 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-30T11:58:39.512 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/32 2023-10-30T11:58:39.512 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-30T11:58:39.512 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/32 2023-10-30T11:58:39.620 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T11:58:39.620 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:58:39.621 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2023-10-30T11:58:39.621 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:39.621 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:39.621 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:58:39.621 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-30T11:58:39.622 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:39.622 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:39.622 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:39.622 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:39.622 INFO:teuthology.orchestra.run.smithi003.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-30T11:58:39.622 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-30T11:58:39.623 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-30T11:58:39.623 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-30T11:58:39.623 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:39.623 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-30T11:58:39.623 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-30T11:58:39.624 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-30T11:58:39.624 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:39.624 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-30T11:58:39.624 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:39.624 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-30T11:58:39.625 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-30T11:58:39.625 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-30T11:58:39.625 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:39.625 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:39.626 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:39.626 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-30T11:58:39.626 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:39.626 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:39.627 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:39.627 INFO:teuthology.orchestra.run.smithi003.stdout: re2-1:20211101-3.el9.x86_64 2023-10-30T11:58:39.627 INFO:teuthology.orchestra.run.smithi003.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-30T11:58:39.627 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:58:39.627 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:58:39.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-test 2023-10-30T11:58:40.540 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:29 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:58:40.569 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T11:58:40.569 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:58:40.569 INFO:teuthology.orchestra.run.smithi092.stdout:Upgraded: 2023-10-30T11:58:40.570 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:40.570 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:40.570 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:58:40.571 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-30T11:58:40.571 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:40.571 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:40.572 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:40.572 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:40.573 INFO:teuthology.orchestra.run.smithi092.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-30T11:58:40.573 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-30T11:58:40.573 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-30T11:58:40.574 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-30T11:58:40.574 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:40.575 INFO:teuthology.orchestra.run.smithi092.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-30T11:58:40.575 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-30T11:58:40.575 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-30T11:58:40.576 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:40.576 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-30T11:58:40.576 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:40.577 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-30T11:58:40.578 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-30T11:58:40.578 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-30T11:58:40.578 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:40.579 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:40.579 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:40.580 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-30T11:58:40.580 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:40.580 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:40.581 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:40.581 INFO:teuthology.orchestra.run.smithi092.stdout: re2-1:20211101-3.el9.x86_64 2023-10-30T11:58:40.583 INFO:teuthology.orchestra.run.smithi092.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-30T11:58:40.583 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:58:40.584 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:58:40.693 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:58:40.694 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:58:40.694 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-30T11:58:40.694 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:58:40.694 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:58:40.695 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 41 M 2023-10-30T11:58:40.695 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-30T11:58:40.695 INFO:teuthology.orchestra.run.smithi003.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-30T11:58:40.695 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 163 k 2023-10-30T11:58:40.696 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-30T11:58:40.696 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-30T11:58:40.696 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-30T11:58:40.697 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:58:40.697 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:58:40.697 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:58:40.697 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2023-10-30T11:58:40.697 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:58:40.698 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 42 M 2023-10-30T11:58:40.698 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 172 M 2023-10-30T11:58:40.699 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:58:40.923 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-test 2023-10-30T11:58:40.974 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2023-10-30T11:58:41.041 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 300 kB 00:00 2023-10-30T11:58:41.075 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-10-30T11:58:41.217 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): libcephsqlite-18.0.0-6839.gcc36dd85.el9. 414 kB/s | 163 kB 00:00 2023-10-30T11:58:41.461 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:31 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:58:41.599 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:58:41.600 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:58:41.600 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-10-30T11:58:41.600 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:58:41.601 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:58:41.601 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 41 M 2023-10-30T11:58:41.601 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-10-30T11:58:41.602 INFO:teuthology.orchestra.run.smithi092.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-30T11:58:41.602 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 163 k 2023-10-30T11:58:41.602 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-30T11:58:41.602 INFO:teuthology.orchestra.run.smithi092.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-30T11:58:41.603 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-30T11:58:41.603 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:58:41.603 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:58:41.604 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:58:41.604 INFO:teuthology.orchestra.run.smithi092.stdout:Install 6 Packages 2023-10-30T11:58:41.604 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:58:41.605 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 42 M 2023-10-30T11:58:41.605 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 172 M 2023-10-30T11:58:41.605 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:58:41.816 INFO:teuthology.orchestra.run.smithi092.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2023-10-30T11:58:41.858 INFO:teuthology.orchestra.run.smithi092.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.4 MB/s | 300 kB 00:00 2023-10-30T11:58:41.860 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): jq-1.6-15.el9.x86_64.rpm 238 kB/s | 187 kB 00:00 2023-10-30T11:58:41.891 INFO:teuthology.orchestra.run.smithi092.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-10-30T11:58:42.008 INFO:teuthology.orchestra.run.smithi092.stdout:(4/6): jq-1.6-15.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2023-10-30T11:58:42.166 INFO:teuthology.orchestra.run.smithi092.stdout:(5/6): libcephsqlite-18.0.0-6839.gcc36dd85.el9. 325 kB/s | 163 kB 00:00 2023-10-30T11:58:43.134 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): ceph-test-18.0.0-6839.gcc36dd85.el9.x86_ 18 MB/s | 41 MB 00:02 2023-10-30T11:58:43.137 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:58:43.138 INFO:teuthology.orchestra.run.smithi003.stdout:Total 17 MB/s | 42 MB 00:02 2023-10-30T11:58:43.152 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:58:43.176 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:58:43.176 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:58:43.559 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:58:43.559 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:58:44.116 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:58:44.194 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-10-30T11:58:44.234 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-10-30T11:58:44.267 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T11:58:44.310 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-10-30T11:58:44.337 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/6 2023-10-30T11:58:46.933 INFO:teuthology.orchestra.run.smithi092.stdout:(6/6): ceph-test-18.0.0-6839.gcc36dd85.el9.x86_ 7.8 MB/s | 41 MB 00:05 2023-10-30T11:58:46.935 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:58:46.936 INFO:teuthology.orchestra.run.smithi092.stdout:Total 7.8 MB/s | 42 MB 00:05 2023-10-30T11:58:46.948 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:58:46.966 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:58:46.967 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:58:47.334 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:58:47.335 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:58:47.883 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:58:47.964 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-10-30T11:58:48.005 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-10-30T11:58:48.041 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T11:58:48.083 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-10-30T11:58:48.111 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/6 2023-10-30T11:58:48.389 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/6 2023-10-30T11:58:48.462 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/6 2023-10-30T11:58:50.553 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/6 2023-10-30T11:58:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/6 2023-10-30T11:58:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/6 2023-10-30T11:58:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T11:58:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-10-30T11:58:50.555 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-10-30T11:58:50.924 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-10-30T11:58:50.924 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:58:50.924 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:58:50.924 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:50.925 INFO:teuthology.orchestra.run.smithi003.stdout: jq-1.6-15.el9.x86_64 2023-10-30T11:58:50.925 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:50.925 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-30T11:58:50.925 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-30T11:58:50.925 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-30T11:58:50.926 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:58:50.926 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:58:51.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph 2023-10-30T11:58:51.735 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:40 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:58:51.885 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:58:51.887 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2023-10-30T11:58:51.888 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-30T11:58:51.888 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2023-10-30T11:58:51.888 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:58:51.888 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 6.5 k 2023-10-30T11:58:51.888 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-30T11:58:51.889 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 2.1 M 2023-10-30T11:58:51.889 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.5 M 2023-10-30T11:58:51.889 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 243 k 2023-10-30T11:58:51.889 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 4.4 M 2023-10-30T11:58:51.889 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 18 M 2023-10-30T11:58:51.890 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-30T11:58:51.890 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-30T11:58:51.890 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-30T11:58:51.890 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-30T11:58:51.891 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-30T11:58:51.891 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-30T11:58:51.891 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-30T11:58:51.891 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-30T11:58:51.891 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-30T11:58:51.892 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-30T11:58:51.892 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-30T11:58:51.892 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-30T11:58:51.892 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-30T11:58:51.892 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-30T11:58:51.892 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-30T11:58:51.893 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-30T11:58:51.893 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-30T11:58:51.893 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-30T11:58:51.893 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-30T11:58:51.893 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-30T11:58:51.894 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-30T11:58:51.894 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-30T11:58:51.894 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-30T11:58:51.894 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-10-30T11:58:51.895 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-30T11:58:51.895 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-30T11:58:51.895 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-30T11:58:51.895 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:58:51.895 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:58:51.896 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2023-10-30T11:58:51.896 INFO:teuthology.orchestra.run.smithi003.stdout:Install 33 Packages 2023-10-30T11:58:51.896 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:58:51.897 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 30 M 2023-10-30T11:58:51.897 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 110 M 2023-10-30T11:58:51.897 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:58:51.956 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/6 2023-10-30T11:58:52.035 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/6 2023-10-30T11:58:52.777 INFO:teuthology.orchestra.run.smithi003.stdout:(1/33): ceph-18.0.0-6839.gcc36dd85.el9.x86_64.r 40 kB/s | 6.5 kB 00:00 2023-10-30T11:58:53.019 INFO:teuthology.orchestra.run.smithi003.stdout:(2/33): ceph-mgr-18.0.0-6839.gcc36dd85.el9.x86_ 3.6 MB/s | 1.5 MB 00:00 2023-10-30T11:58:53.186 INFO:teuthology.orchestra.run.smithi003.stdout:(3/33): ceph-mon-18.0.0-6839.gcc36dd85.el9.x86_ 11 MB/s | 4.4 MB 00:00 2023-10-30T11:58:53.261 INFO:teuthology.orchestra.run.smithi003.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6839.gcc36 3.2 MB/s | 243 kB 00:00 2023-10-30T11:58:53.353 INFO:teuthology.orchestra.run.smithi003.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 144 kB 00:00 2023-10-30T11:58:53.378 INFO:teuthology.orchestra.run.smithi003.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2023-10-30T11:58:53.403 INFO:teuthology.orchestra.run.smithi003.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2023-10-30T11:58:53.445 INFO:teuthology.orchestra.run.smithi003.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-10-30T11:58:53.479 INFO:teuthology.orchestra.run.smithi003.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-10-30T11:58:53.504 INFO:teuthology.orchestra.run.smithi003.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 4.9 MB/s | 124 kB 00:00 2023-10-30T11:58:53.538 INFO:teuthology.orchestra.run.smithi003.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-10-30T11:58:53.571 INFO:teuthology.orchestra.run.smithi003.stdout:(12/33): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-10-30T11:58:53.755 INFO:teuthology.orchestra.run.smithi003.stdout:(13/33): python3-cryptography-36.0.1-4.el9.x86_ 6.7 MB/s | 1.2 MB 00:00 2023-10-30T11:58:53.930 INFO:teuthology.orchestra.run.smithi003.stdout:(14/33): ceph-osd-18.0.0-6839.gcc36dd85.el9.x86 19 MB/s | 18 MB 00:00 2023-10-30T11:58:53.956 INFO:teuthology.orchestra.run.smithi003.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 254 kB/s | 51 kB 00:00 2023-10-30T11:58:53.997 INFO:teuthology.orchestra.run.smithi003.stdout:(16/33): ceph-mds-18.0.0-6839.gcc36dd85.el9.x86 1.5 MB/s | 2.1 MB 00:01 2023-10-30T11:58:54.116 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 6/6 2023-10-30T11:58:54.116 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/6 2023-10-30T11:58:54.116 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/6 2023-10-30T11:58:54.116 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T11:58:54.117 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-10-30T11:58:54.117 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-10-30T11:58:54.406 INFO:teuthology.orchestra.run.smithi003.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 91 kB/s | 43 kB 00:00 2023-10-30T11:58:54.506 INFO:teuthology.orchestra.run.smithi003.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2023-10-30T11:58:54.515 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-10-30T11:58:54.515 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:58:54.515 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:58:54.515 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:54.516 INFO:teuthology.orchestra.run.smithi092.stdout: jq-1.6-15.el9.x86_64 2023-10-30T11:58:54.516 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:58:54.516 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-30T11:58:54.516 INFO:teuthology.orchestra.run.smithi092.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-30T11:58:54.516 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-30T11:58:54.516 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:58:54.517 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:58:54.531 INFO:teuthology.orchestra.run.smithi003.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 298 kB/s | 172 kB 00:00 2023-10-30T11:58:54.615 INFO:teuthology.orchestra.run.smithi003.stdout:(20/33): python3-cherrypy-18.6.1-2.el9.noarch.r 581 kB/s | 358 kB 00:00 2023-10-30T11:58:54.641 INFO:teuthology.orchestra.run.smithi003.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 132 kB/s | 18 kB 00:00 2023-10-30T11:58:54.674 INFO:teuthology.orchestra.run.smithi003.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 163 kB/s | 23 kB 00:00 2023-10-30T11:58:54.716 INFO:teuthology.orchestra.run.smithi003.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 193 kB/s | 19 kB 00:00 2023-10-30T11:58:54.775 INFO:teuthology.orchestra.run.smithi003.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2023-10-30T11:58:54.799 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph 2023-10-30T11:58:54.800 INFO:teuthology.orchestra.run.smithi003.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 369 kB/s | 46 kB 00:00 2023-10-30T11:58:54.825 INFO:teuthology.orchestra.run.smithi003.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 723 kB/s | 79 kB 00:00 2023-10-30T11:58:54.900 INFO:teuthology.orchestra.run.smithi003.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2023-10-30T11:58:54.934 INFO:teuthology.orchestra.run.smithi003.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 828 kB/s | 90 kB 00:00 2023-10-30T11:58:54.993 INFO:teuthology.orchestra.run.smithi003.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 388 kB/s | 36 kB 00:00 2023-10-30T11:58:55.043 INFO:teuthology.orchestra.run.smithi003.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.0 MB/s | 272 kB 00:00 2023-10-30T11:58:55.068 INFO:teuthology.orchestra.run.smithi003.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2023-10-30T11:58:55.135 INFO:teuthology.orchestra.run.smithi003.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2023-10-30T11:58:55.349 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:45 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:58:55.452 INFO:teuthology.orchestra.run.smithi003.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 931 kB/s | 427 kB 00:00 2023-10-30T11:58:55.458 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:58:55.459 INFO:teuthology.orchestra.run.smithi003.stdout:Total 8.4 MB/s | 30 MB 00:03 2023-10-30T11:58:55.492 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:58:55.499 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:58:55.501 INFO:teuthology.orchestra.run.smithi092.stdout:========================================================================================= 2023-10-30T11:58:55.502 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-10-30T11:58:55.502 INFO:teuthology.orchestra.run.smithi092.stdout:========================================================================================= 2023-10-30T11:58:55.502 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:58:55.502 INFO:teuthology.orchestra.run.smithi092.stdout: ceph x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 6.5 k 2023-10-30T11:58:55.502 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-10-30T11:58:55.503 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 2.1 M 2023-10-30T11:58:55.503 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.5 M 2023-10-30T11:58:55.503 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 243 k 2023-10-30T11:58:55.503 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 4.4 M 2023-10-30T11:58:55.503 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 18 M 2023-10-30T11:58:55.504 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-30T11:58:55.504 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-30T11:58:55.504 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-30T11:58:55.504 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-30T11:58:55.504 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-30T11:58:55.505 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-30T11:58:55.505 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-30T11:58:55.505 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-30T11:58:55.505 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-30T11:58:55.505 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-30T11:58:55.506 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-30T11:58:55.506 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-30T11:58:55.506 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-30T11:58:55.506 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-30T11:58:55.506 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-30T11:58:55.507 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-30T11:58:55.507 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-30T11:58:55.507 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-30T11:58:55.507 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-30T11:58:55.507 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-30T11:58:55.508 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-30T11:58:55.508 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-30T11:58:55.508 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-30T11:58:55.508 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-10-30T11:58:55.508 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-30T11:58:55.509 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-30T11:58:55.509 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-30T11:58:55.509 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:58:55.509 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:58:55.509 INFO:teuthology.orchestra.run.smithi092.stdout:========================================================================================= 2023-10-30T11:58:55.510 INFO:teuthology.orchestra.run.smithi092.stdout:Install 33 Packages 2023-10-30T11:58:55.510 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:58:55.510 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 30 M 2023-10-30T11:58:55.511 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 110 M 2023-10-30T11:58:55.511 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:58:55.536 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:58:55.536 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:58:55.908 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:58:55.908 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:58:56.378 INFO:teuthology.orchestra.run.smithi092.stdout:(1/33): ceph-18.0.0-6839.gcc36dd85.el9.x86_64.r 35 kB/s | 6.5 kB 00:00 2023-10-30T11:58:56.559 INFO:teuthology.orchestra.run.smithi092.stdout:(2/33): ceph-mds-18.0.0-6839.gcc36dd85.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2023-10-30T11:58:56.617 INFO:teuthology.orchestra.run.smithi092.stdout:(3/33): ceph-mgr-18.0.0-6839.gcc36dd85.el9.x86_ 3.4 MB/s | 1.5 MB 00:00 2023-10-30T11:58:56.710 INFO:teuthology.orchestra.run.smithi092.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6839.gcc36 2.6 MB/s | 243 kB 00:00 2023-10-30T11:58:56.801 INFO:teuthology.orchestra.run.smithi092.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 144 kB 00:00 2023-10-30T11:58:56.827 INFO:teuthology.orchestra.run.smithi092.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2023-10-30T11:58:56.860 INFO:teuthology.orchestra.run.smithi092.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-10-30T11:58:56.902 INFO:teuthology.orchestra.run.smithi092.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-10-30T11:58:56.936 INFO:teuthology.orchestra.run.smithi092.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-10-30T11:58:56.969 INFO:teuthology.orchestra.run.smithi092.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-10-30T11:58:57.003 INFO:teuthology.orchestra.run.smithi092.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-10-30T11:58:57.037 INFO:teuthology.orchestra.run.smithi092.stdout:(12/33): python3-urllib3-1.26.5-4.el9.noarch.rp 5.4 MB/s | 187 kB 00:00 2023-10-30T11:58:57.204 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:58:57.255 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-30T11:58:57.287 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-30T11:58:57.303 INFO:teuthology.orchestra.run.smithi092.stdout:(13/33): python3-cryptography-36.0.1-4.el9.x86_ 4.6 MB/s | 1.2 MB 00:00 2023-10-30T11:58:57.320 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-30T11:58:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-30T11:58:57.479 INFO:teuthology.orchestra.run.smithi092.stdout:(14/33): ceph-osd-18.0.0-6839.gcc36dd85.el9.x86 19 MB/s | 18 MB 00:00 2023-10-30T11:58:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-30T11:58:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-30T11:58:57.554 INFO:teuthology.orchestra.run.smithi092.stdout:(15/33): ceph-mon-18.0.0-6839.gcc36dd85.el9.x86 3.7 MB/s | 4.4 MB 00:01 2023-10-30T11:58:57.578 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-30T11:58:57.604 INFO:teuthology.orchestra.run.smithi092.stdout:(16/33): python3-pytz-2021.1-5.el9.noarch.rpm 170 kB/s | 51 kB 00:00 2023-10-30T11:58:57.656 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-30T11:58:57.716 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-10-30T11:58:57.754 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-30T11:58:57.763 INFO:teuthology.orchestra.run.smithi092.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 153 kB/s | 43 kB 00:00 2023-10-30T11:58:57.812 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T11:58:57.838 INFO:teuthology.orchestra.run.smithi092.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 143 kB/s | 11 kB 00:00 2023-10-30T11:58:57.874 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T11:58:57.921 INFO:teuthology.orchestra.run.smithi092.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 213 kB/s | 18 kB 00:00 2023-10-30T11:58:57.947 INFO:teuthology.orchestra.run.smithi092.stdout:(20/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 467 kB/s | 172 kB 00:00 2023-10-30T11:58:58.005 INFO:teuthology.orchestra.run.smithi092.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 276 kB/s | 23 kB 00:00 2023-10-30T11:58:58.031 INFO:teuthology.orchestra.run.smithi092.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 233 kB/s | 19 kB 00:00 2023-10-30T11:58:58.056 INFO:teuthology.orchestra.run.smithi092.stdout:(23/33): python3-cherrypy-18.6.1-2.el9.noarch.r 794 kB/s | 358 kB 00:00 2023-10-30T11:58:58.076 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-30T11:58:58.097 INFO:teuthology.orchestra.run.smithi092.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2023-10-30T11:58:58.123 INFO:teuthology.orchestra.run.smithi092.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 503 kB/s | 46 kB 00:00 2023-10-30T11:58:58.124 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-30T11:58:58.148 INFO:teuthology.orchestra.run.smithi092.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 855 kB/s | 79 kB 00:00 2023-10-30T11:58:58.163 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-30T11:58:58.198 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-30T11:58:58.215 INFO:teuthology.orchestra.run.smithi092.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 178 kB/s | 16 kB 00:00 2023-10-30T11:58:58.234 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-30T11:58:58.240 INFO:teuthology.orchestra.run.smithi092.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 978 kB/s | 90 kB 00:00 2023-10-30T11:58:58.307 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-30T11:58:58.344 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-30T11:58:58.377 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-30T11:58:58.420 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-30T11:58:58.424 INFO:teuthology.orchestra.run.smithi092.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 171 kB/s | 36 kB 00:00 2023-10-30T11:58:58.449 INFO:teuthology.orchestra.run.smithi092.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 775 kB/s | 272 kB 00:00 2023-10-30T11:58:58.453 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-30T11:58:58.491 INFO:teuthology.orchestra.run.smithi092.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 916 kB/s | 230 kB 00:00 2023-10-30T11:58:58.525 INFO:teuthology.orchestra.run.smithi092.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2023-10-30T11:58:58.548 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-30T11:58:58.575 INFO:teuthology.orchestra.run.smithi092.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2023-10-30T11:58:58.581 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:58:58.581 INFO:teuthology.orchestra.run.smithi092.stdout:Total 9.7 MB/s | 30 MB 00:03 2023-10-30T11:58:58.584 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-30T11:58:58.613 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:58:58.651 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:58:58.651 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:58:58.659 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-30T11:58:58.803 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-30T11:58:58.934 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T11:58:59.026 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:58:59.027 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:58:59.095 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9. 28/33 2023-10-30T11:58:59.122 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/33 2023-10-30T11:58:59.150 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/33 2023-10-30T11:58:59.150 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:58:59.150 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-30T11:58:59.150 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T11:58:59.151 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T11:58:59.151 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:59:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-30T11:59:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-30T11:59:00.368 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-30T11:59:00.459 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-30T11:59:00.528 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-30T11:59:00.570 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-30T11:59:00.608 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-30T11:59:00.683 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-30T11:59:00.746 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-10-30T11:59:00.748 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/33 2023-10-30T11:59:00.776 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/33 2023-10-30T11:59:00.776 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:00.776 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-30T11:59:00.777 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T11:59:00.777 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T11:59:00.777 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:00.785 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-30T11:59:00.833 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T11:59:00.898 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T11:59:01.090 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-30T11:59:01.139 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-30T11:59:01.178 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-30T11:59:01.213 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-30T11:59:01.240 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/33 2023-10-30T11:59:01.242 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-30T11:59:01.264 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/33 2023-10-30T11:59:01.265 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:01.265 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-30T11:59:01.265 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T11:59:01.265 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T11:59:01.266 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:01.318 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-30T11:59:01.360 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-30T11:59:01.393 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-30T11:59:01.441 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-30T11:59:01.469 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-30T11:59:01.474 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 32/33 2023-10-30T11:59:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 32/33 2023-10-30T11:59:01.499 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:01.499 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-30T11:59:01.499 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T11:59:01.499 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T11:59:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:01.557 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 33/33 2023-10-30T11:59:01.557 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-30T11:59:01.600 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-30T11:59:01.680 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-30T11:59:01.824 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-30T11:59:01.970 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T11:59:02.133 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9. 28/33 2023-10-30T11:59:02.163 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/33 2023-10-30T11:59:02.188 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 29/33 2023-10-30T11:59:02.189 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:02.189 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-30T11:59:02.189 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T11:59:02.190 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T11:59:02.190 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:03.276 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 33/33 2023-10-30T11:59:03.277 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/33 2023-10-30T11:59:03.277 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/33 2023-10-30T11:59:03.277 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 3/33 2023-10-30T11:59:03.277 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 4/33 2023-10-30T11:59:03.277 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/33 2023-10-30T11:59:03.277 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9. 6/33 2023-10-30T11:59:03.278 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-30T11:59:03.278 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-30T11:59:03.278 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-30T11:59:03.278 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-30T11:59:03.278 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T11:59:03.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T11:59:03.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-30T11:59:03.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-10-30T11:59:03.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-30T11:59:03.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-30T11:59:03.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-30T11:59:03.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-30T11:59:03.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-30T11:59:03.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-30T11:59:03.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-30T11:59:03.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-30T11:59:03.281 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-30T11:59:03.281 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-30T11:59:03.281 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-30T11:59:03.282 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-30T11:59:03.282 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T11:59:03.282 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-30T11:59:03.283 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-30T11:59:03.283 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-30T11:59:03.283 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-30T11:59:03.283 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-30T11:59:03.697 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/33 2023-10-30T11:59:03.723 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 30/33 2023-10-30T11:59:03.723 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:03.723 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-30T11:59:03.723 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T11:59:03.724 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T11:59:03.724 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:04.115 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-30T11:59:04.115 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:04.115 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:59:04.115 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:04.116 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:04.116 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:04.116 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:04.116 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:04.116 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:04.117 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-30T11:59:04.117 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-30T11:59:04.117 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-30T11:59:04.117 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-30T11:59:04.117 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-30T11:59:04.117 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-30T11:59:04.118 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-30T11:59:04.118 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-30T11:59:04.118 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-30T11:59:04.118 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-30T11:59:04.119 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-30T11:59:04.119 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-30T11:59:04.119 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-30T11:59:04.119 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-30T11:59:04.119 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-30T11:59:04.119 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply-3.11-14.el9.noarch 2023-10-30T11:59:04.120 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-30T11:59:04.120 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-30T11:59:04.120 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-30T11:59:04.120 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-30T11:59:04.120 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-30T11:59:04.121 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-30T11:59:04.121 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-30T11:59:04.121 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-10-30T11:59:04.121 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-30T11:59:04.121 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-30T11:59:04.122 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-30T11:59:04.122 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:04.122 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:04.164 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/33 2023-10-30T11:59:04.187 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 31/33 2023-10-30T11:59:04.187 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:04.187 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-30T11:59:04.188 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T11:59:04.188 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T11:59:04.188 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:04.390 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 32/33 2023-10-30T11:59:04.412 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 32/33 2023-10-30T11:59:04.412 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:04.413 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-30T11:59:04.413 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T11:59:04.413 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T11:59:04.413 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:04.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-base 2023-10-30T11:59:04.471 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 33/33 2023-10-30T11:59:05.011 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:54 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:05.104 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:05.154 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:05.155 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-30T11:59:05.156 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:05.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install cephadm 2023-10-30T11:59:05.802 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:54 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:05.948 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:05.949 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-30T11:59:05.949 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:05.949 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:59:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 270 k 2023-10-30T11:59:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:05.950 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:59:05.950 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:05.950 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T11:59:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:05.951 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 270 k 2023-10-30T11:59:05.951 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 265 k 2023-10-30T11:59:05.951 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:59:06.210 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 33/33 2023-10-30T11:59:06.210 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/33 2023-10-30T11:59:06.210 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/33 2023-10-30T11:59:06.211 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 3/33 2023-10-30T11:59:06.211 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 4/33 2023-10-30T11:59:06.211 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 5/33 2023-10-30T11:59:06.211 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9. 6/33 2023-10-30T11:59:06.211 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-30T11:59:06.212 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-30T11:59:06.212 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-30T11:59:06.212 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-30T11:59:06.212 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T11:59:06.212 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T11:59:06.213 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-30T11:59:06.213 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-10-30T11:59:06.213 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-30T11:59:06.213 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-30T11:59:06.213 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-30T11:59:06.213 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-30T11:59:06.214 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-30T11:59:06.214 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-30T11:59:06.214 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-30T11:59:06.214 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-30T11:59:06.214 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-30T11:59:06.215 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-30T11:59:06.215 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-30T11:59:06.216 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-30T11:59:06.216 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T11:59:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-30T11:59:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-30T11:59:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-30T11:59:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-30T11:59:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-30T11:59:06.472 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm-18.0.0-6839.gcc36dd85.el9.noarch.rpm 516 kB/s | 270 kB 00:00 2023-10-30T11:59:06.473 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:06.473 INFO:teuthology.orchestra.run.smithi003.stdout:Total 514 kB/s | 270 kB 00:00 2023-10-30T11:59:06.473 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:59:06.479 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:59:06.479 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:59:06.482 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:59:06.483 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:59:06.557 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:59:07.013 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T11:59:07.096 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T11:59:07.104 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-30T11:59:07.104 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:07.105 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:59:07.105 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:07.105 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:07.105 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:07.106 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:07.106 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:07.106 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:07.106 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-30T11:59:07.106 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-30T11:59:07.107 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-30T11:59:07.107 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-30T11:59:07.107 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-30T11:59:07.107 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-30T11:59:07.107 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-30T11:59:07.108 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-30T11:59:07.108 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-30T11:59:07.108 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-30T11:59:07.108 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-30T11:59:07.108 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-30T11:59:07.109 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-30T11:59:07.109 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-30T11:59:07.109 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-30T11:59:07.109 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply-3.11-14.el9.noarch 2023-10-30T11:59:07.109 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-30T11:59:07.110 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-30T11:59:07.110 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-30T11:59:07.110 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-30T11:59:07.110 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-30T11:59:07.110 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-30T11:59:07.111 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-30T11:59:07.111 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-10-30T11:59:07.111 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-30T11:59:07.111 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-30T11:59:07.111 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-30T11:59:07.111 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:07.112 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:07.437 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T11:59:07.443 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-base 2023-10-30T11:59:07.738 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T11:59:07.738 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:07.738 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:59:07.738 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:07.739 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:07.739 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:07.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-immutable-object-cache 2023-10-30T11:59:07.993 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:57 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:08.083 INFO:teuthology.orchestra.run.smithi092.stdout:Package ceph-base-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:08.132 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:08.133 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-10-30T11:59:08.133 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:08.190 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install cephadm 2023-10-30T11:59:08.521 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:57 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:08.667 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:08.668 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:08.668 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T11:59:08.668 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:08.668 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:59:08.669 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 141 k 2023-10-30T11:59:08.669 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:08.669 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:59:08.669 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:08.669 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T11:59:08.670 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:08.670 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 141 k 2023-10-30T11:59:08.670 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 395 k 2023-10-30T11:59:08.670 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:59:08.730 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:58 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:08.869 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:08.870 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:08.870 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-10-30T11:59:08.870 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:08.871 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:59:08.871 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 270 k 2023-10-30T11:59:08.871 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:08.871 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:59:08.871 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:08.872 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T11:59:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:08.872 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 270 k 2023-10-30T11:59:08.872 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 265 k 2023-10-30T11:59:08.872 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:59:08.885 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-immutable-object-cache-18.0.0-6839.gcc36dd 649 kB/s | 141 kB 00:00 2023-10-30T11:59:08.886 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:08.886 INFO:teuthology.orchestra.run.smithi003.stdout:Total 645 kB/s | 141 kB 00:00 2023-10-30T11:59:08.887 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:59:08.895 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:59:08.895 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:59:08.950 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:59:08.950 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:59:09.102 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:59:09.178 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T11:59:09.204 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T11:59:09.204 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:09.204 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-30T11:59:09.205 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:09.288 INFO:teuthology.orchestra.run.smithi092.stdout:cephadm-18.0.0-6839.gcc36dd85.el9.noarch.rpm 648 kB/s | 270 kB 00:00 2023-10-30T11:59:09.288 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:09.289 INFO:teuthology.orchestra.run.smithi092.stdout:Total 645 kB/s | 270 kB 00:00 2023-10-30T11:59:09.289 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:59:09.293 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:59:09.294 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:59:09.296 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:59:09.297 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:59:09.365 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:59:09.726 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T11:59:09.817 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T11:59:10.169 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T11:59:10.170 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:10.170 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:59:10.170 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:10.170 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:10.171 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:10.184 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T11:59:10.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr 2023-10-30T11:59:10.498 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T11:59:10.498 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:10.499 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:59:10.499 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:10.499 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:10.499 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:10.695 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-immutable-object-cache 2023-10-30T11:59:11.002 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:00 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:11.095 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:11.147 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:11.148 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-30T11:59:11.149 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:11.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-dashboard 2023-10-30T11:59:11.255 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:01 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:11.398 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:11.399 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:11.399 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T11:59:11.400 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:11.400 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:59:11.400 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 141 k 2023-10-30T11:59:11.400 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:11.401 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:59:11.401 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:11.401 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T11:59:11.401 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:11.402 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 141 k 2023-10-30T11:59:11.402 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 395 k 2023-10-30T11:59:11.402 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:59:11.601 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-immutable-object-cache-18.0.0-6839.gcc36dd 700 kB/s | 141 kB 00:00 2023-10-30T11:59:11.602 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:11.602 INFO:teuthology.orchestra.run.smithi092.stdout:Total 695 kB/s | 141 kB 00:00 2023-10-30T11:59:11.603 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:59:11.610 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:59:11.610 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:59:11.661 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:59:11.662 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:59:11.786 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:00 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:11.822 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:59:11.886 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T11:59:11.910 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T11:59:11.911 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:11.911 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-30T11:59:11.911 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:11.931 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:11.932 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:11.932 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-30T11:59:11.932 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:11.933 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:59:11.933 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 1.8 M 2023-10-30T11:59:11.933 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-30T11:59:11.933 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 23 k 2023-10-30T11:59:11.934 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 15 k 2023-10-30T11:59:11.934 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-30T11:59:11.934 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-30T11:59:11.934 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-30T11:59:11.935 INFO:teuthology.orchestra.run.smithi003.stdout:Installing weak dependencies: 2023-10-30T11:59:11.935 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-30T11:59:11.935 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:11.935 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:59:11.936 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:11.936 INFO:teuthology.orchestra.run.smithi003.stdout:Install 7 Packages 2023-10-30T11:59:11.936 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:11.937 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 2.1 M 2023-10-30T11:59:11.937 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 8.6 M 2023-10-30T11:59:11.938 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:59:12.189 INFO:teuthology.orchestra.run.smithi003.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6839.gcc3 119 kB/s | 23 kB 00:00 2023-10-30T11:59:12.206 INFO:teuthology.orchestra.run.smithi003.stdout:(2/7): ceph-prometheus-alerts-18.0.0-6839.gcc36 71 kB/s | 15 kB 00:00 2023-10-30T11:59:12.431 INFO:teuthology.orchestra.run.smithi003.stdout:(3/7): ceph-mgr-dashboard-18.0.0-6839.gcc36dd85 4.0 MB/s | 1.8 MB 00:00 2023-10-30T11:59:12.448 INFO:teuthology.orchestra.run.smithi003.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 35 kB/s | 9.0 kB 00:00 2023-10-30T11:59:12.523 INFO:teuthology.orchestra.run.smithi003.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 129 kB/s | 41 kB 00:00 2023-10-30T11:59:12.573 INFO:teuthology.orchestra.run.smithi003.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 217 kB/s | 31 kB 00:00 2023-10-30T11:59:12.765 INFO:teuthology.orchestra.run.smithi003.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 595 kB/s | 188 kB 00:00 2023-10-30T11:59:12.767 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:12.767 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.5 MB/s | 2.1 MB 00:00 2023-10-30T11:59:12.768 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:59:12.777 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:59:12.777 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:59:12.849 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:59:12.850 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:59:12.911 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85. 1/1 2023-10-30T11:59:12.912 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:12.912 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:59:12.912 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:12.912 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:12.913 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:13.136 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr 2023-10-30T11:59:13.158 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:59:13.225 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-30T11:59:13.259 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-30T11:59:13.331 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-30T11:59:13.371 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-30T11:59:13.399 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.n 5/7 2023-10-30T11:59:13.652 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9. 6/7 2023-10-30T11:59:13.677 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 7/7 2023-10-30T11:59:13.697 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:03 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:13.786 INFO:teuthology.orchestra.run.smithi092.stdout:Package ceph-mgr-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:13.835 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:13.835 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-10-30T11:59:13.836 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:13.890 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-dashboard 2023-10-30T11:59:14.268 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 7/7 2023-10-30T11:59:14.268 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9. 1/7 2023-10-30T11:59:14.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 2/7 2023-10-30T11:59:14.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.n 3/7 2023-10-30T11:59:14.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-30T11:59:14.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-30T11:59:14.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-30T11:59:14.430 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:04 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:14.568 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:14.569 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:14.569 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-10-30T11:59:14.569 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:14.569 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:59:14.569 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 1.8 M 2023-10-30T11:59:14.570 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-10-30T11:59:14.570 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 23 k 2023-10-30T11:59:14.570 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 15 k 2023-10-30T11:59:14.570 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-30T11:59:14.571 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-30T11:59:14.571 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-30T11:59:14.571 INFO:teuthology.orchestra.run.smithi092.stdout:Installing weak dependencies: 2023-10-30T11:59:14.571 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-30T11:59:14.571 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:14.571 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:59:14.572 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:14.572 INFO:teuthology.orchestra.run.smithi092.stdout:Install 7 Packages 2023-10-30T11:59:14.572 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:14.573 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 2.1 M 2023-10-30T11:59:14.573 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 8.6 M 2023-10-30T11:59:14.573 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:59:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-30T11:59:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:14.591 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:59:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-30T11:59:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-30T11:59:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-30T11:59:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-30T11:59:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:14.593 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:14.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-30T11:59:14.921 INFO:teuthology.orchestra.run.smithi092.stdout:(1/7): ceph-prometheus-alerts-18.0.0-6839.gcc36 82 kB/s | 15 kB 00:00 2023-10-30T11:59:14.996 INFO:teuthology.orchestra.run.smithi092.stdout:(2/7): ceph-grafana-dashboards-18.0.0-6839.gcc3 90 kB/s | 23 kB 00:00 2023-10-30T11:59:15.063 INFO:teuthology.orchestra.run.smithi092.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 64 kB/s | 9.0 kB 00:00 2023-10-30T11:59:15.097 INFO:teuthology.orchestra.run.smithi092.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 412 kB/s | 41 kB 00:00 2023-10-30T11:59:15.122 INFO:teuthology.orchestra.run.smithi092.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 527 kB/s | 31 kB 00:00 2023-10-30T11:59:15.164 INFO:teuthology.orchestra.run.smithi092.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2023-10-30T11:59:15.339 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:04 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:15.430 INFO:teuthology.orchestra.run.smithi092.stdout:(7/7): ceph-mgr-dashboard-18.0.0-6839.gcc36dd85 2.5 MB/s | 1.8 MB 00:00 2023-10-30T11:59:15.434 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:15.434 INFO:teuthology.orchestra.run.smithi092.stdout:Total 2.4 MB/s | 2.1 MB 00:00 2023-10-30T11:59:15.434 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:59:15.442 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:59:15.442 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:59:15.485 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:15.486 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2023-10-30T11:59:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-30T11:59:15.487 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2023-10-30T11:59:15.487 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:59:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 7.4 M 2023-10-30T11:59:15.488 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-30T11:59:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-30T11:59:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-30T11:59:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-30T11:59:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-30T11:59:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-30T11:59:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-30T11:59:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-30T11:59:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-30T11:59:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-30T11:59:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-30T11:59:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-30T11:59:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:15.490 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:59:15.490 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2023-10-30T11:59:15.491 INFO:teuthology.orchestra.run.smithi003.stdout:Install 12 Packages 2023-10-30T11:59:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:15.491 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 37 M 2023-10-30T11:59:15.492 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 209 M 2023-10-30T11:59:15.492 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:59:15.508 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:59:15.508 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:59:15.621 INFO:teuthology.orchestra.run.smithi003.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 240 kB/s | 32 kB 00:00 2023-10-30T11:59:15.655 INFO:teuthology.orchestra.run.smithi003.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 532 kB/s | 17 kB 00:00 2023-10-30T11:59:15.688 INFO:teuthology.orchestra.run.smithi003.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-10-30T11:59:15.755 INFO:teuthology.orchestra.run.smithi003.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-10-30T11:59:15.798 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:59:15.864 INFO:teuthology.orchestra.run.smithi003.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2023-10-30T11:59:15.865 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-30T11:59:15.889 INFO:teuthology.orchestra.run.smithi003.stdout:(6/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 205 kB 00:00 2023-10-30T11:59:15.899 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-30T11:59:15.945 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-30T11:59:15.965 INFO:teuthology.orchestra.run.smithi003.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 48 MB/s | 4.8 MB 00:00 2023-10-30T11:59:15.977 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-30T11:59:15.990 INFO:teuthology.orchestra.run.smithi003.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-10-30T11:59:16.006 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.n 5/7 2023-10-30T11:59:16.198 INFO:teuthology.orchestra.run.smithi003.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 67 MB/s | 16 MB 00:00 2023-10-30T11:59:16.224 INFO:teuthology.orchestra.run.smithi003.stdout:(10/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 3.4 MB/s | 803 kB 00:00 2023-10-30T11:59:16.242 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9. 6/7 2023-10-30T11:59:16.249 INFO:teuthology.orchestra.run.smithi003.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.9 MB/s | 195 kB 00:00 2023-10-30T11:59:16.284 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 7/7 2023-10-30T11:59:16.844 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 7/7 2023-10-30T11:59:16.844 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9. 1/7 2023-10-30T11:59:16.844 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarc 2/7 2023-10-30T11:59:16.845 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.n 3/7 2023-10-30T11:59:16.845 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-30T11:59:16.845 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-30T11:59:16.845 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-30T11:59:16.860 INFO:teuthology.orchestra.run.smithi003.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 5.4 MB/s | 7.4 MB 00:01 2023-10-30T11:59:16.863 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:16.864 INFO:teuthology.orchestra.run.smithi003.stdout:Total 27 MB/s | 37 MB 00:01 2023-10-30T11:59:16.864 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:59:16.889 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:59:16.889 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:59:17.153 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-30T11:59:17.153 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:17.154 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:59:17.154 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:17.154 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:17.154 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:17.154 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-30T11:59:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-30T11:59:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-30T11:59:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-30T11:59:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:17.155 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:17.343 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-30T11:59:17.352 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:59:17.353 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:59:17.891 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:07 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:18.031 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:18.032 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2023-10-30T11:59:18.032 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-10-30T11:59:18.032 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2023-10-30T11:59:18.033 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:59:18.033 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 7.4 M 2023-10-30T11:59:18.033 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-10-30T11:59:18.033 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-30T11:59:18.033 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-30T11:59:18.034 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-30T11:59:18.034 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-30T11:59:18.034 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-30T11:59:18.034 INFO:teuthology.orchestra.run.smithi092.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-30T11:59:18.034 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-30T11:59:18.035 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-30T11:59:18.035 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-30T11:59:18.035 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-30T11:59:18.035 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-30T11:59:18.035 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:18.035 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:59:18.036 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2023-10-30T11:59:18.036 INFO:teuthology.orchestra.run.smithi092.stdout:Install 12 Packages 2023-10-30T11:59:18.036 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:18.037 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 37 M 2023-10-30T11:59:18.037 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 209 M 2023-10-30T11:59:18.037 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:59:18.162 INFO:teuthology.orchestra.run.smithi092.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 248 kB/s | 32 kB 00:00 2023-10-30T11:59:18.199 INFO:teuthology.orchestra.run.smithi092.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 528 kB/s | 17 kB 00:00 2023-10-30T11:59:18.200 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:59:18.229 INFO:teuthology.orchestra.run.smithi092.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-10-30T11:59:18.274 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-30T11:59:18.288 INFO:teuthology.orchestra.run.smithi092.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-10-30T11:59:18.304 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-30T11:59:18.380 INFO:teuthology.orchestra.run.smithi092.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 32 MB/s | 4.7 MB 00:00 2023-10-30T11:59:18.402 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-30T11:59:18.405 INFO:teuthology.orchestra.run.smithi092.stdout:(6/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 205 kB 00:00 2023-10-30T11:59:18.438 INFO:teuthology.orchestra.run.smithi092.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 6.4 MB/s | 217 kB 00:00 2023-10-30T11:59:18.440 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-30T11:59:18.514 INFO:teuthology.orchestra.run.smithi092.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 44 MB/s | 4.8 MB 00:00 2023-10-30T11:59:18.628 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T11:59:18.689 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T11:59:18.706 INFO:teuthology.orchestra.run.smithi092.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 59 MB/s | 16 MB 00:00 2023-10-30T11:59:18.731 INFO:teuthology.orchestra.run.smithi092.stdout:(10/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 3.7 MB/s | 803 kB 00:00 2023-10-30T11:59:18.756 INFO:teuthology.orchestra.run.smithi092.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.9 MB/s | 195 kB 00:00 2023-10-30T11:59:18.775 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-30T11:59:19.348 INFO:teuthology.orchestra.run.smithi092.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 5.6 MB/s | 7.4 MB 00:01 2023-10-30T11:59:19.350 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:19.351 INFO:teuthology.orchestra.run.smithi092.stdout:Total 28 MB/s | 37 MB 00:01 2023-10-30T11:59:19.351 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:59:19.372 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:59:19.373 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:59:19.575 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-30T11:59:19.680 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T11:59:19.810 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:59:19.811 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:59:20.518 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:59:20.613 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-30T11:59:20.652 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-30T11:59:20.743 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-30T11:59:20.788 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-30T11:59:20.957 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T11:59:20.984 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-30T11:59:20.987 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T11:59:21.070 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-30T11:59:21.722 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-30T11:59:21.832 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T11:59:22.068 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-30T11:59:22.104 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 12/12 2023-10-30T11:59:23.051 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-30T11:59:24.086 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-30T11:59:24.112 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 12/12 2023-10-30T11:59:24.735 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 12/12 2023-10-30T11:59:24.735 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 1/12 2023-10-30T11:59:24.735 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-30T11:59:24.736 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-30T11:59:24.736 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-30T11:59:24.736 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T11:59:24.736 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T11:59:24.736 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-30T11:59:24.737 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-30T11:59:24.737 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T11:59:24.737 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-30T11:59:24.737 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-30T11:59:25.225 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-30T11:59:25.225 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:25.225 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:59:25.226 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:25.226 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-30T11:59:25.226 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-30T11:59:25.226 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-30T11:59:25.226 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-30T11:59:25.226 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-30T11:59:25.227 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-30T11:59:25.227 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-30T11:59:25.227 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-30T11:59:25.227 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-30T11:59:25.227 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-30T11:59:25.228 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-30T11:59:25.228 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:25.228 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:25.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-rook 2023-10-30T11:59:26.172 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:15 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:26.321 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:26.322 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2023-10-30T11:59:26.322 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-30T11:59:26.322 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2023-10-30T11:59:26.322 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:59:26.322 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 48 k 2023-10-30T11:59:26.323 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-30T11:59:26.323 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-30T11:59:26.323 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-30T11:59:26.323 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-30T11:59:26.324 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-30T11:59:26.324 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-30T11:59:26.324 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-30T11:59:26.324 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-30T11:59:26.324 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-30T11:59:26.325 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-30T11:59:26.325 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-30T11:59:26.325 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-30T11:59:26.325 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-30T11:59:26.325 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:26.326 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:59:26.326 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2023-10-30T11:59:26.326 INFO:teuthology.orchestra.run.smithi003.stdout:Install 13 Packages 2023-10-30T11:59:26.326 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:26.327 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 2.0 M 2023-10-30T11:59:26.327 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 26 M 2023-10-30T11:59:26.328 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:59:26.535 INFO:teuthology.orchestra.run.smithi003.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 188 kB/s | 24 kB 00:00 2023-10-30T11:59:26.561 INFO:teuthology.orchestra.run.smithi003.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 18 kB 00:00 2023-10-30T11:59:26.574 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 12/12 2023-10-30T11:59:26.574 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36d 1/12 2023-10-30T11:59:26.575 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-30T11:59:26.575 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-30T11:59:26.575 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-30T11:59:26.575 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T11:59:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T11:59:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-30T11:59:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-30T11:59:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T11:59:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-30T11:59:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-30T11:59:26.602 INFO:teuthology.orchestra.run.smithi003.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-10-30T11:59:26.628 INFO:teuthology.orchestra.run.smithi003.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-10-30T11:59:26.653 INFO:teuthology.orchestra.run.smithi003.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-30T11:59:26.679 INFO:teuthology.orchestra.run.smithi003.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2023-10-30T11:59:26.729 INFO:teuthology.orchestra.run.smithi003.stdout:(7/13): ceph-mgr-rook-18.0.0-6839.gcc36dd85.el9 148 kB/s | 48 kB 00:00 2023-10-30T11:59:26.896 INFO:teuthology.orchestra.run.smithi003.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 133 kB/s | 32 kB 00:00 2023-10-30T11:59:26.912 INFO:teuthology.orchestra.run.smithi003.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2023-10-30T11:59:27.013 INFO:teuthology.orchestra.run.smithi003.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 590 kB/s | 59 kB 00:00 2023-10-30T11:59:27.043 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-30T11:59:27.043 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:27.044 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:59:27.044 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:27.044 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-30T11:59:27.044 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-30T11:59:27.044 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-30T11:59:27.045 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-30T11:59:27.045 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-30T11:59:27.045 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-30T11:59:27.045 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-30T11:59:27.046 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-30T11:59:27.046 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-30T11:59:27.046 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-30T11:59:27.046 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-30T11:59:27.046 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:27.047 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:27.063 INFO:teuthology.orchestra.run.smithi003.stdout:(11/13): python3-google-auth-2.23.3-1.el9.noarc 689 kB/s | 197 kB 00:00 2023-10-30T11:59:27.105 INFO:teuthology.orchestra.run.smithi003.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 977 kB/s | 90 kB 00:00 2023-10-30T11:59:27.155 INFO:teuthology.orchestra.run.smithi003.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2023-10-30T11:59:27.159 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:27.159 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2023-10-30T11:59:27.160 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:59:27.173 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:59:27.173 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:59:27.279 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:59:27.280 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:59:27.377 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-rook 2023-10-30T11:59:27.819 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:59:27.940 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:17 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:27.987 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-30T11:59:28.042 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-30T11:59:28.093 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:28.094 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-10-30T11:59:28.095 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-10-30T11:59:28.095 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-10-30T11:59:28.095 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:59:28.095 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 48 k 2023-10-30T11:59:28.096 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-10-30T11:59:28.096 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-30T11:59:28.096 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-30T11:59:28.096 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-30T11:59:28.096 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-30T11:59:28.097 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-30T11:59:28.097 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-30T11:59:28.097 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-30T11:59:28.097 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-30T11:59:28.098 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-30T11:59:28.098 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-30T11:59:28.098 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-30T11:59:28.098 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-30T11:59:28.098 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:28.099 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:59:28.099 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-10-30T11:59:28.099 INFO:teuthology.orchestra.run.smithi092.stdout:Install 13 Packages 2023-10-30T11:59:28.099 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:28.100 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 2.0 M 2023-10-30T11:59:28.100 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 26 M 2023-10-30T11:59:28.100 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:59:28.101 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-30T11:59:28.135 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-30T11:59:28.177 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-30T11:59:28.273 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-30T11:59:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-10-30T11:59:28.454 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-30T11:59:28.669 INFO:teuthology.orchestra.run.smithi092.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 215 kB/s | 24 kB 00:00 2023-10-30T11:59:28.694 INFO:teuthology.orchestra.run.smithi092.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 130 kB/s | 18 kB 00:00 2023-10-30T11:59:28.736 INFO:teuthology.orchestra.run.smithi092.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-10-30T11:59:28.761 INFO:teuthology.orchestra.run.smithi092.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-10-30T11:59:28.787 INFO:teuthology.orchestra.run.smithi092.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-30T11:59:28.812 INFO:teuthology.orchestra.run.smithi092.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2023-10-30T11:59:28.837 INFO:teuthology.orchestra.run.smithi092.stdout:(7/13): ceph-mgr-rook-18.0.0-6839.gcc36dd85.el9 169 kB/s | 48 kB 00:00 2023-10-30T11:59:28.987 INFO:teuthology.orchestra.run.smithi092.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 160 kB/s | 32 kB 00:00 2023-10-30T11:59:29.013 INFO:teuthology.orchestra.run.smithi092.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 75 kB/s | 15 kB 00:00 2023-10-30T11:59:29.080 INFO:teuthology.orchestra.run.smithi092.stdout:(10/13): python3-google-auth-2.23.3-1.el9.noarc 1.0 MB/s | 197 kB 00:00 2023-10-30T11:59:29.136 INFO:teuthology.orchestra.run.smithi092.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 479 kB/s | 59 kB 00:00 2023-10-30T11:59:29.172 INFO:teuthology.orchestra.run.smithi092.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 5.6 MB/s | 1.0 MB 00:00 2023-10-30T11:59:29.197 INFO:teuthology.orchestra.run.smithi092.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 771 kB/s | 90 kB 00:00 2023-10-30T11:59:29.201 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:29.202 INFO:teuthology.orchestra.run.smithi092.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2023-10-30T11:59:29.202 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:59:29.215 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:59:29.215 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:59:29.219 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-30T11:59:29.266 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-30T11:59:29.300 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-30T11:59:29.309 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:59:29.310 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:59:29.346 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-30T11:59:29.369 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 13/13 2023-10-30T11:59:29.831 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:59:29.992 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-30T11:59:30.048 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-30T11:59:30.103 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-30T11:59:30.132 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-30T11:59:30.164 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 13/13 2023-10-30T11:59:30.164 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 1/13 2023-10-30T11:59:30.164 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-30T11:59:30.165 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-30T11:59:30.165 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-30T11:59:30.165 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-30T11:59:30.165 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-30T11:59:30.165 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-30T11:59:30.166 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-30T11:59:30.166 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-30T11:59:30.166 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-10-30T11:59:30.166 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-30T11:59:30.166 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-30T11:59:30.173 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-30T11:59:30.265 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-30T11:59:30.383 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-10-30T11:59:30.433 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-30T11:59:30.550 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-30T11:59:30.550 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:30.551 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:59:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-30T11:59:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-30T11:59:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-30T11:59:30.552 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-30T11:59:30.552 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-30T11:59:30.552 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-30T11:59:30.552 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-30T11:59:30.552 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-30T11:59:30.552 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-30T11:59:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-30T11:59:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-30T11:59:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-30T11:59:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:30.553 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:30.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-cephadm 2023-10-30T11:59:31.149 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-30T11:59:31.197 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-30T11:59:31.230 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-30T11:59:31.277 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-30T11:59:31.300 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 13/13 2023-10-30T11:59:31.352 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:20 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:31.498 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:31.499 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2023-10-30T11:59:31.499 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-30T11:59:31.499 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2023-10-30T11:59:31.499 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:59:31.500 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 132 k 2023-10-30T11:59:31.500 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-30T11:59:31.500 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-30T11:59:31.500 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-30T11:59:31.501 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-30T11:59:31.501 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-30T11:59:31.501 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-30T11:59:31.501 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:31.501 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:59:31.502 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2023-10-30T11:59:31.502 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2023-10-30T11:59:31.502 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:31.503 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 6.8 M 2023-10-30T11:59:31.503 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 33 M 2023-10-30T11:59:31.503 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:59:31.742 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-10-30T11:59:31.884 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2023-10-30T11:59:32.001 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): ceph-mgr-cephadm-18.0.0-6839.gcc36dd85.e 325 kB/s | 132 kB 00:00 2023-10-30T11:59:32.051 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 13/13 2023-10-30T11:59:32.051 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 1/13 2023-10-30T11:59:32.052 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-30T11:59:32.052 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-30T11:59:32.052 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-30T11:59:32.052 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-30T11:59:32.053 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-30T11:59:32.053 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-30T11:59:32.053 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-30T11:59:32.053 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-30T11:59:32.053 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-10-30T11:59:32.054 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-30T11:59:32.054 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-30T11:59:32.218 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 173 kB/s | 58 kB 00:00 2023-10-30T11:59:32.326 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 157 kB/s | 51 kB 00:00 2023-10-30T11:59:32.393 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 844 kB/s | 548 kB 00:00 2023-10-30T11:59:32.395 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:32.396 INFO:teuthology.orchestra.run.smithi003.stdout:Total 7.6 MB/s | 6.8 MB 00:00 2023-10-30T11:59:32.396 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:59:32.407 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:59:32.407 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:59:32.411 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-30T11:59:32.411 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:32.411 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:59:32.412 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:32.412 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-30T11:59:32.412 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-30T11:59:32.413 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-30T11:59:32.413 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-30T11:59:32.413 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-30T11:59:32.413 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-30T11:59:32.413 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-30T11:59:32.413 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-30T11:59:32.414 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-30T11:59:32.414 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-30T11:59:32.414 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-30T11:59:32.414 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-30T11:59:32.414 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:32.415 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:32.507 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:59:32.507 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:59:32.650 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-cephadm 2023-10-30T11:59:32.813 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:59:32.983 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-30T11:59:33.029 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-30T11:59:33.193 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:23 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:33.336 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:33.336 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-10-30T11:59:33.337 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-10-30T11:59:33.337 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-10-30T11:59:33.337 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:59:33.337 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 132 k 2023-10-30T11:59:33.337 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-10-30T11:59:33.338 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-30T11:59:33.338 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-30T11:59:33.338 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-30T11:59:33.338 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-30T11:59:33.338 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-30T11:59:33.339 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:33.339 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:59:33.339 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-10-30T11:59:33.339 INFO:teuthology.orchestra.run.smithi092.stdout:Install 6 Packages 2023-10-30T11:59:33.339 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:33.340 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 6.8 M 2023-10-30T11:59:33.340 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 33 M 2023-10-30T11:59:33.340 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:59:33.539 INFO:teuthology.orchestra.run.smithi092.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-10-30T11:59:33.592 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-30T11:59:33.664 INFO:teuthology.orchestra.run.smithi092.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2023-10-30T11:59:33.683 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-30T11:59:33.723 INFO:teuthology.orchestra.run.smithi092.stdout:(3/6): ceph-mgr-cephadm-18.0.0-6839.gcc36dd85.e 399 kB/s | 132 kB 00:00 2023-10-30T11:59:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-30T11:59:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 6/6 2023-10-30T11:59:34.307 INFO:teuthology.orchestra.run.smithi092.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 93 kB/s | 58 kB 00:00 2023-10-30T11:59:34.357 INFO:teuthology.orchestra.run.smithi092.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 80 kB/s | 51 kB 00:00 2023-10-30T11:59:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 6/6 2023-10-30T11:59:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/6 2023-10-30T11:59:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-30T11:59:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-30T11:59:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-30T11:59:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-30T11:59:34.848 INFO:teuthology.orchestra.run.smithi092.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 419 kB/s | 548 kB 00:01 2023-10-30T11:59:34.853 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:34.853 INFO:teuthology.orchestra.run.smithi092.stdout:Total 4.5 MB/s | 6.8 MB 00:01 2023-10-30T11:59:34.853 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:59:34.863 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:59:34.863 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:59:34.938 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-30T11:59:34.939 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:34.939 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:59:34.939 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:34.939 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-30T11:59:34.939 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-30T11:59:34.940 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-30T11:59:34.940 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-30T11:59:34.940 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-30T11:59:34.940 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:34.940 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:34.953 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:59:34.953 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:59:35.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-fuse 2023-10-30T11:59:35.253 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:59:35.420 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-30T11:59:35.475 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-30T11:59:35.798 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:24 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:35.943 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:35.944 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:35.944 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-30T11:59:35.944 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:35.945 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:59:35.945 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 855 k 2023-10-30T11:59:35.945 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-30T11:59:35.945 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-30T11:59:35.945 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:35.946 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:59:35.946 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:35.946 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2023-10-30T11:59:35.946 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:35.947 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 934 k 2023-10-30T11:59:35.947 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 2.7 M 2023-10-30T11:59:35.947 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:59:35.980 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-30T11:59:36.061 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 683 kB/s | 79 kB 00:00 2023-10-30T11:59:36.090 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-30T11:59:36.155 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-30T11:59:36.179 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 6/6 2023-10-30T11:59:36.386 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): ceph-fuse-18.0.0-6839.gcc36dd85.el9.x86_ 1.9 MB/s | 855 kB 00:00 2023-10-30T11:59:36.388 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:36.388 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.1 MB/s | 934 kB 00:00 2023-10-30T11:59:36.388 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:59:36.411 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:59:36.411 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:59:36.496 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:59:36.496 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:59:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:59:36.840 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-30T11:59:36.905 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T11:59:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 6/6 2023-10-30T11:59:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 1/6 2023-10-30T11:59:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-30T11:59:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-30T11:59:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-30T11:59:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-30T11:59:37.298 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-30T11:59:37.299 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:37.299 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:59:37.299 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:37.299 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-30T11:59:37.299 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-30T11:59:37.300 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-30T11:59:37.300 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-30T11:59:37.300 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-30T11:59:37.300 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:37.300 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:37.535 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-fuse 2023-10-30T11:59:37.718 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T11:59:37.719 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/2 2023-10-30T11:59:38.050 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-30T11:59:38.050 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:38.050 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:59:38.051 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-30T11:59:38.051 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:38.051 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:38.081 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:28 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:38.221 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:38.222 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:38.222 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-10-30T11:59:38.223 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:38.223 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:59:38.223 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 855 k 2023-10-30T11:59:38.223 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-10-30T11:59:38.223 INFO:teuthology.orchestra.run.smithi092.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-30T11:59:38.224 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:38.224 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:59:38.224 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:38.224 INFO:teuthology.orchestra.run.smithi092.stdout:Install 2 Packages 2023-10-30T11:59:38.224 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:38.224 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 934 k 2023-10-30T11:59:38.225 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 2.7 M 2023-10-30T11:59:38.225 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:59:38.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-volume 2023-10-30T11:59:38.341 INFO:teuthology.orchestra.run.smithi092.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 668 kB/s | 79 kB 00:00 2023-10-30T11:59:38.741 INFO:teuthology.orchestra.run.smithi092.stdout:(2/2): ceph-fuse-18.0.0-6839.gcc36dd85.el9.x86_ 1.6 MB/s | 855 kB 00:00 2023-10-30T11:59:38.743 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:38.743 INFO:teuthology.orchestra.run.smithi092.stdout:Total 1.8 MB/s | 934 kB 00:00 2023-10-30T11:59:38.743 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:59:38.763 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:59:38.763 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:59:38.842 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:59:38.842 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:59:38.869 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:27 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:39.016 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:39.016 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:39.016 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-30T11:59:39.017 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:39.017 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:59:39.017 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 262 k 2023-10-30T11:59:39.017 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:39.018 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:59:39.018 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:39.018 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T11:59:39.018 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:39.019 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 262 k 2023-10-30T11:59:39.019 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 1.2 M 2023-10-30T11:59:39.020 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:59:39.045 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:59:39.189 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-30T11:59:39.248 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-volume-18.0.0-6839.gcc36dd85.el9.noarch.rp 1.1 MB/s | 262 kB 00:00 2023-10-30T11:59:39.249 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:39.249 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.1 MB/s | 262 kB 00:00 2023-10-30T11:59:39.250 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:59:39.258 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:59:39.258 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:59:39.265 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T11:59:39.294 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:59:39.295 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:59:39.519 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:59:39.584 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T11:59:39.600 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T11:59:39.600 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:39.601 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-30T11:59:39.601 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:39.912 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T11:59:39.913 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/2 2023-10-30T11:59:40.206 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-30T11:59:40.206 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:40.206 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:59:40.206 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-30T11:59:40.207 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:40.207 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:40.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-volume 2023-10-30T11:59:40.686 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T11:59:40.686 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:40.686 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:59:40.687 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:40.687 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:40.687 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:40.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados-devel 2023-10-30T11:59:41.012 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:31 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:41.154 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:41.155 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:41.155 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-10-30T11:59:41.155 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:41.155 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:59:41.155 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume noarch 2:18.0.0-6839.gcc36dd85.el9 ceph-noarch 262 k 2023-10-30T11:59:41.156 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:41.156 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:59:41.156 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:41.156 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T11:59:41.156 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:41.157 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 262 k 2023-10-30T11:59:41.157 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 1.2 M 2023-10-30T11:59:41.157 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:59:41.579 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:30 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:41.727 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:41.727 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:41.728 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T11:59:41.728 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:41.728 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:59:41.728 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 127 k 2023-10-30T11:59:41.729 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:41.729 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:59:41.729 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:41.729 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T11:59:41.729 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:41.730 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 127 k 2023-10-30T11:59:41.730 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 456 k 2023-10-30T11:59:41.730 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:59:41.830 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-volume-18.0.0-6839.gcc36dd85.el9.noarch.rp 389 kB/s | 262 kB 00:00 2023-10-30T11:59:41.831 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:41.831 INFO:teuthology.orchestra.run.smithi092.stdout:Total 388 kB/s | 262 kB 00:00 2023-10-30T11:59:41.831 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:59:41.839 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:59:41.839 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:59:41.877 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:59:41.878 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:59:42.091 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:59:42.157 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T11:59:42.173 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T11:59:42.173 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:42.173 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-30T11:59:42.174 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:42.587 INFO:teuthology.orchestra.run.smithi003.stdout:librados-devel-18.0.0-6839.gcc36dd85.el9.x86_64 148 kB/s | 127 kB 00:00 2023-10-30T11:59:42.588 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:42.588 INFO:teuthology.orchestra.run.smithi003.stdout:Total 147 kB/s | 127 kB 00:00 2023-10-30T11:59:42.588 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:59:42.594 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:59:42.595 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:59:42.658 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:59:42.659 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:59:42.812 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:59:42.921 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:43.204 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 1/1 2023-10-30T11:59:43.204 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:43.205 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:59:43.205 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume-2:18.0.0-6839.gcc36dd85.el9.noarch 2023-10-30T11:59:43.205 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:43.205 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:43.370 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:43.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librados-devel 2023-10-30T11:59:43.678 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:43.678 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:43.678 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:59:43.679 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:43.679 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:43.679 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:43.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs2 2023-10-30T11:59:44.107 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:34 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:44.248 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:44.249 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:44.249 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T11:59:44.249 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:44.250 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:59:44.250 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 127 k 2023-10-30T11:59:44.250 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:44.250 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:59:44.251 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:44.251 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T11:59:44.251 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:44.251 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 127 k 2023-10-30T11:59:44.251 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 456 k 2023-10-30T11:59:44.251 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:59:44.501 INFO:teuthology.orchestra.run.smithi092.stdout:librados-devel-18.0.0-6839.gcc36dd85.el9.x86_64 504 kB/s | 127 kB 00:00 2023-10-30T11:59:44.502 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:44.502 INFO:teuthology.orchestra.run.smithi092.stdout:Total 501 kB/s | 127 kB 00:00 2023-10-30T11:59:44.502 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:59:44.507 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:59:44.507 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:59:44.521 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:33 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:44.562 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:59:44.562 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:59:44.614 INFO:teuthology.orchestra.run.smithi003.stdout:Package libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:44.667 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:44.667 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-30T11:59:44.668 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:44.704 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:59:44.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs-devel 2023-10-30T11:59:44.799 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:45.248 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:45.319 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:34 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:45.465 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:45.466 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:45.466 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T11:59:45.467 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:45.467 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:59:45.467 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 31 k 2023-10-30T11:59:45.467 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:45.467 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:59:45.467 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:45.468 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T11:59:45.468 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:45.468 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 31 k 2023-10-30T11:59:45.468 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 139 k 2023-10-30T11:59:45.468 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:59:45.529 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:45.529 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:45.529 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:59:45.529 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:45.529 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:45.530 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:45.736 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install libcephfs2 2023-10-30T11:59:45.739 INFO:teuthology.orchestra.run.smithi003.stdout:libcephfs-devel-18.0.0-6839.gcc36dd85.el9.x86_6 114 kB/s | 31 kB 00:00 2023-10-30T11:59:45.740 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:45.740 INFO:teuthology.orchestra.run.smithi003.stdout:Total 114 kB/s | 31 kB 00:00 2023-10-30T11:59:45.741 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:59:45.744 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:59:45.744 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:59:45.768 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:59:45.768 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:59:45.858 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:59:45.964 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:46.288 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:36 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:46.380 INFO:teuthology.orchestra.run.smithi092.stdout:Package libcephfs2-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:46.385 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:46.432 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:46.433 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-10-30T11:59:46.433 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:46.489 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install libcephfs-devel 2023-10-30T11:59:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:46.660 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:59:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:46.660 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:46.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados2 2023-10-30T11:59:47.034 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:37 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:47.176 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:47.177 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:47.177 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T11:59:47.177 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:47.177 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:59:47.177 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 31 k 2023-10-30T11:59:47.178 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:47.178 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:59:47.178 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:47.178 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T11:59:47.178 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:47.179 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 31 k 2023-10-30T11:59:47.179 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 139 k 2023-10-30T11:59:47.179 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:59:47.346 INFO:teuthology.orchestra.run.smithi092.stdout:libcephfs-devel-18.0.0-6839.gcc36dd85.el9.x86_6 185 kB/s | 31 kB 00:00 2023-10-30T11:59:47.346 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:47.347 INFO:teuthology.orchestra.run.smithi092.stdout:Total 184 kB/s | 31 kB 00:00 2023-10-30T11:59:47.347 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:59:47.350 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:59:47.350 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:59:47.381 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:59:47.381 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:59:47.422 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:36 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:47.471 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:59:47.516 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:47.569 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:47.570 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-30T11:59:47.570 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:47.584 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:47.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librbd1 2023-10-30T11:59:47.963 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:48.222 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:48.223 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:48.223 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:59:48.223 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:48.223 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:48.224 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:48.228 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:37 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:48.323 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:48.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librados2 2023-10-30T11:59:48.375 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:48.377 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-30T11:59:48.377 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:48.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rados 2023-10-30T11:59:48.923 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:38 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:49.014 INFO:teuthology.orchestra.run.smithi092.stdout:Package librados2-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:49.028 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:38 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:49.064 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:49.065 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-10-30T11:59:49.065 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:49.121 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:49.125 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librbd1 2023-10-30T11:59:49.174 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:49.175 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-30T11:59:49.176 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:49.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rgw 2023-10-30T11:59:49.675 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:39 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:49.766 INFO:teuthology.orchestra.run.smithi092.stdout:Package librbd1-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:49.816 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:49.816 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-10-30T11:59:49.817 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:49.835 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:38 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:49.883 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rados 2023-10-30T11:59:49.928 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:49.981 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:49.982 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-30T11:59:49.982 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:50.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-cephfs 2023-10-30T11:59:50.427 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:40 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:50.516 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-rados-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:50.565 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:50.566 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-10-30T11:59:50.567 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:50.621 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rgw 2023-10-30T11:59:50.629 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:39 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:50.721 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:50.774 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:50.775 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-30T11:59:50.776 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:50.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rbd 2023-10-30T11:59:51.164 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:41 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:51.256 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-rgw-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:51.306 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:51.307 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-10-30T11:59:51.307 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:51.366 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-cephfs 2023-10-30T11:59:51.427 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:40 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:51.521 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:51.573 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:51.574 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-30T11:59:51.574 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:51.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-fuse 2023-10-30T11:59:51.909 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:41 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:51.998 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-cephfs-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:52.048 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:52.049 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-10-30T11:59:52.049 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:52.104 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rbd 2023-10-30T11:59:52.240 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:41 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:52.386 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:52.387 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-10-30T11:59:52.388 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:52.388 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:59:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 85 k 2023-10-30T11:59:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:52.388 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:59:52.388 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:52.389 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T11:59:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:52.389 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 85 k 2023-10-30T11:59:52.389 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 227 k 2023-10-30T11:59:52.389 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:59:52.652 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:42 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:52.742 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-rbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T11:59:52.743 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-fuse-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 239 kB/s | 85 kB 00:00 2023-10-30T11:59:52.743 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:52.744 INFO:teuthology.orchestra.run.smithi003.stdout:Total 238 kB/s | 85 kB 00:00 2023-10-30T11:59:52.744 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:59:52.751 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:59:52.752 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:59:52.792 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:52.793 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-10-30T11:59:52.793 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:52.809 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:59:52.810 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:59:52.852 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-fuse 2023-10-30T11:59:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:59:53.064 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:53.396 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:43 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:53.484 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:53.536 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:53.537 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:53.537 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2023-10-30T11:59:53.538 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:53.538 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:59:53.538 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 85 k 2023-10-30T11:59:53.538 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:53.538 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:59:53.539 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:53.539 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T11:59:53.539 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:53.539 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 85 k 2023-10-30T11:59:53.539 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 227 k 2023-10-30T11:59:53.540 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:59:53.801 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:53.801 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:53.802 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:59:53.802 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:53.802 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:53.802 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:53.848 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-fuse-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 274 kB/s | 85 kB 00:00 2023-10-30T11:59:53.849 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:53.849 INFO:teuthology.orchestra.run.smithi092.stdout:Total 273 kB/s | 85 kB 00:00 2023-10-30T11:59:53.849 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:59:53.856 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:59:53.856 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:59:53.911 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:59:53.911 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:59:54.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-mirror 2023-10-30T11:59:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:59:54.154 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:54.520 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:54.607 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:43 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:54.755 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:54.756 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:54.757 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T11:59:54.757 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:54.757 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:59:54.757 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.0 M 2023-10-30T11:59:54.757 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:54.758 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:59:54.758 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:54.758 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T11:59:54.758 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:54.759 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 3.0 M 2023-10-30T11:59:54.759 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 12 M 2023-10-30T11:59:54.759 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:59:54.811 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:54.811 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:54.811 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:59:54.811 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:54.812 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:54.812 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:55.053 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-mirror 2023-10-30T11:59:55.420 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-mirror-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 4.6 MB/s | 3.0 MB 00:00 2023-10-30T11:59:55.422 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:55.422 INFO:teuthology.orchestra.run.smithi003.stdout:Total 4.5 MB/s | 3.0 MB 00:00 2023-10-30T11:59:55.422 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:59:55.432 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:59:55.432 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:59:55.510 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:59:55.510 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:59:55.614 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:45 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T11:59:55.759 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T11:59:55.760 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:55.760 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T11:59:55.760 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:55.760 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T11:59:55.760 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 3.0 M 2023-10-30T11:59:55.761 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:55.761 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T11:59:55.761 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T11:59:55.761 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T11:59:55.761 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:55.762 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 3.0 M 2023-10-30T11:59:55.762 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 12 M 2023-10-30T11:59:55.762 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T11:59:55.946 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:59:56.013 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:56.037 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:56.038 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:56.038 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-30T11:59:56.038 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T11:59:56.038 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T11:59:56.039 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:57.187 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:57.188 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:57.188 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:59:57.188 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:57.188 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:57.189 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:57.459 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-mirror-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 1.8 MB/s | 3.0 MB 00:01 2023-10-30T11:59:57.460 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:57.460 INFO:teuthology.orchestra.run.smithi092.stdout:Total 1.8 MB/s | 3.0 MB 00:01 2023-10-30T11:59:57.461 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T11:59:57.470 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T11:59:57.471 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T11:59:57.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-nbd 2023-10-30T11:59:57.545 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T11:59:57.546 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T11:59:57.957 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T11:59:58.018 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:58.042 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:58.043 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:58.043 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-30T11:59:58.043 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T11:59:58.043 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T11:59:58.043 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:58.059 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:47 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T11:59:58.206 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T11:59:58.207 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:58.207 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-10-30T11:59:58.208 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:58.208 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T11:59:58.208 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 168 k 2023-10-30T11:59:58.208 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:58.209 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T11:59:58.209 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T11:59:58.209 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T11:59:58.209 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:58.209 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 168 k 2023-10-30T11:59:58.210 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 486 k 2023-10-30T11:59:58.210 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T11:59:58.415 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-nbd-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 814 kB/s | 168 kB 00:00 2023-10-30T11:59:58.416 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:58.416 INFO:teuthology.orchestra.run.smithi003.stdout:Total 808 kB/s | 168 kB 00:00 2023-10-30T11:59:58.416 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T11:59:58.424 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T11:59:58.424 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T11:59:58.484 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T11:59:58.485 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T11:59:58.638 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T11:59:58.741 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:59.163 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T11:59:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T11:59:59.164 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T11:59:59.223 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:59.411 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-nbd 2023-10-30T11:59:59.532 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T11:59:59.532 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:59.533 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T11:59:59.533 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T11:59:59.533 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T11:59:59.534 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T11:59:59.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-base-debuginfo 2023-10-30T11:59:59.984 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:49 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:00:00.124 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:00:00.125 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:00:00.125 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2023-10-30T12:00:00.125 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:00:00.125 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:00:00.126 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 168 k 2023-10-30T12:00:00.126 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:00:00.126 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:00:00.126 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:00:00.127 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:00:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:00:00.127 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 168 k 2023-10-30T12:00:00.127 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 486 k 2023-10-30T12:00:00.127 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:00:00.355 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:49 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:00:00.502 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:00:00.503 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:00:00.503 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:00:00.504 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:00:00.504 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:00:00.504 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 74 M 2023-10-30T12:00:00.504 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-30T12:00:00.504 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 20 M 2023-10-30T12:00:00.505 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:00:00.505 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:00:00.505 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:00:00.505 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2023-10-30T12:00:00.505 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:00:00.505 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 94 M 2023-10-30T12:00:00.506 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 511 M 2023-10-30T12:00:00.506 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:00:00.554 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-nbd-18.0.0-6839.gcc36dd85.el9.x86_64.rpm 393 kB/s | 168 kB 00:00 2023-10-30T12:00:00.555 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:00:00.555 INFO:teuthology.orchestra.run.smithi092.stdout:Total 392 kB/s | 168 kB 00:00 2023-10-30T12:00:00.555 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:00:00.562 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:00:00.562 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:00:00.618 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:00:00.618 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:00:00.764 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:00:00.877 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:00:01.301 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:00:01.613 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:00:01.613 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:00:01.614 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:00:01.614 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:00:01.614 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:00:01.615 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:00:01.867 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-base-debuginfo 2023-10-30T12:00:02.168 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): ceph-debuginfo-18.0.0-6839.gcc36dd85.el9 12 MB/s | 20 MB 00:01 2023-10-30T12:00:02.428 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:52 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:00:02.572 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:00:02.573 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:00:02.573 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:00:02.573 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:00:02.574 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:00:02.574 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 74 M 2023-10-30T12:00:02.574 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-10-30T12:00:02.574 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 20 M 2023-10-30T12:00:02.575 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:00:02.575 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:00:02.575 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:00:02.575 INFO:teuthology.orchestra.run.smithi092.stdout:Install 2 Packages 2023-10-30T12:00:02.575 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:00:02.576 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 94 M 2023-10-30T12:00:02.576 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 511 M 2023-10-30T12:00:02.576 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:00:06.832 INFO:teuthology.orchestra.run.smithi092.stdout:(1/2): ceph-debuginfo-18.0.0-6839.gcc36dd85.el9 4.7 MB/s | 20 MB 00:04 2023-10-30T12:00:09.101 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): ceph-base-debuginfo-18.0.0-6839.gcc36dd8 8.6 MB/s | 74 MB 00:08 2023-10-30T12:00:09.103 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:00:09.103 INFO:teuthology.orchestra.run.smithi003.stdout:Total 11 MB/s | 94 MB 00:08 2023-10-30T12:00:09.104 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:00:09.109 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:00:09.109 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:00:09.761 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:00:09.762 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:00:11.781 INFO:teuthology.orchestra.run.smithi092.stdout:(2/2): ceph-base-debuginfo-18.0.0-6839.gcc36dd8 8.0 MB/s | 74 MB 00:09 2023-10-30T12:00:11.782 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:00:11.783 INFO:teuthology.orchestra.run.smithi092.stdout:Total 10 MB/s | 94 MB 00:09 2023-10-30T12:00:11.783 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:00:11.788 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:00:11.789 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:00:12.414 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:00:12.414 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:00:12.816 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:00:15.297 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:00:20.974 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/2 2023-10-30T12:00:21.045 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 2/2 2023-10-30T12:00:22.962 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/2 2023-10-30T12:00:23.030 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 2/2 2023-10-30T12:00:24.398 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 2/2 2023-10-30T12:00:24.398 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/2 2023-10-30T12:00:24.688 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T12:00:24.688 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:00:24.688 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:00:24.689 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:00:24.689 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:00:24.689 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:00:24.689 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:00:24.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-common-debuginfo 2023-10-30T12:00:25.493 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:14 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:00:25.640 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:00:25.641 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:00:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:00:25.642 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:00:25.642 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:00:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 410 M 2023-10-30T12:00:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:00:25.642 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:00:25.643 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:00:25.643 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:00:25.643 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:00:25.643 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 410 M 2023-10-30T12:00:25.643 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 1.7 G 2023-10-30T12:00:25.643 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:00:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 2/2 2023-10-30T12:00:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/2 2023-10-30T12:00:26.704 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2/2 2023-10-30T12:00:26.704 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:00:26.704 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:00:26.705 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:00:26.705 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:00:26.705 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:00:26.705 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:00:26.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-common-debuginfo 2023-10-30T12:00:27.438 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:02:17 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:00:27.578 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:00:27.579 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:00:27.579 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:00:27.579 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:00:27.579 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:00:27.580 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 410 M 2023-10-30T12:00:27.580 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:00:27.580 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:00:27.580 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:00:27.580 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:00:27.580 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:00:27.581 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 410 M 2023-10-30T12:00:27.581 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 1.7 G 2023-10-30T12:00:27.581 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:00:48.686 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-common-debuginfo-18.0.0-6839.gcc36dd85.el9 18 MB/s | 410 MB 00:23 2023-10-30T12:00:48.687 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:00:48.687 INFO:teuthology.orchestra.run.smithi003.stdout:Total 18 MB/s | 410 MB 00:23 2023-10-30T12:00:48.688 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:00:48.692 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:00:48.692 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:00:51.268 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:00:51.268 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:01:03.145 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-common-debuginfo-18.0.0-6839.gcc36dd85.el9 12 MB/s | 410 MB 00:35 2023-10-30T12:01:03.146 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:01:03.146 INFO:teuthology.orchestra.run.smithi092.stdout:Total 12 MB/s | 410 MB 00:35 2023-10-30T12:01:03.146 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:01:03.152 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:01:03.152 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:01:05.635 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:01:05.636 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:01:35.207 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:01:35.300 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:01:42.950 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:01:43.215 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:01:43.216 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:01:43.216 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:01:43.216 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:01:43.216 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:01:43.217 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:01:43.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-10-30T12:01:44.044 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:03:33 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:01:44.192 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:01:44.193 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2023-10-30T12:01:44.193 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:01:44.193 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2023-10-30T12:01:44.193 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:01:44.194 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.4 M 2023-10-30T12:01:44.194 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:01:44.194 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:01:44.194 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2023-10-30T12:01:44.194 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:01:44.195 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:01:44.195 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 1.4 M 2023-10-30T12:01:44.195 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 6.3 M 2023-10-30T12:01:44.195 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:01:44.760 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-68 2.5 MB/s | 1.4 MB 00:00 2023-10-30T12:01:44.760 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:01:44.761 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.5 MB/s | 1.4 MB 00:00 2023-10-30T12:01:44.761 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:01:44.764 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:01:44.765 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:01:44.811 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:01:44.812 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:01:45.080 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:01:45.185 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T12:01:45.780 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T12:01:46.069 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T12:01:46.069 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:01:46.070 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:01:46.070 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:01:46.070 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:01:46.070 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:01:46.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw-debuginfo 2023-10-30T12:01:46.964 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:03:35 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:01:47.118 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:01:47.124 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:01:47.124 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:01:47.125 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:01:47.125 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:01:47.125 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:01:47.125 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 174 M 2023-10-30T12:01:47.125 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:01:47.126 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:01:47.126 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:01:47.126 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:01:47.126 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:01:47.127 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 174 M 2023-10-30T12:01:47.127 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 732 M 2023-10-30T12:01:47.127 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:01:47.212 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:01:54.950 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:01:55.206 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:01:55.206 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:01:55.206 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:01:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:01:55.207 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:01:55.207 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:01:55.409 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-10-30T12:01:55.949 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:03:45 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:01:56.090 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:01:56.091 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-10-30T12:01:56.091 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:01:56.091 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-10-30T12:01:56.091 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:01:56.092 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.4 M 2023-10-30T12:01:56.092 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:01:56.092 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:01:56.092 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-10-30T12:01:56.092 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:01:56.092 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:01:56.093 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 1.4 M 2023-10-30T12:01:56.093 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 6.3 M 2023-10-30T12:01:56.093 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:01:57.273 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-68 1.2 MB/s | 1.4 MB 00:01 2023-10-30T12:01:57.274 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:01:57.274 INFO:teuthology.orchestra.run.smithi092.stdout:Total 1.2 MB/s | 1.4 MB 00:01 2023-10-30T12:01:57.275 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:01:57.277 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:01:57.278 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:01:57.321 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:01:57.321 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:01:57.640 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:01:57.739 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T12:01:58.125 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T12:01:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-6839. 1/1 2023-10-30T12:01:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:01:58.402 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:01:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:01:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:01:58.403 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:01:58.545 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-radosgw-debuginfo 2023-10-30T12:01:59.088 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:03:49 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:01:59.227 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:01:59.229 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:01:59.229 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:01:59.229 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:01:59.229 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:01:59.230 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 174 M 2023-10-30T12:01:59.230 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:01:59.230 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:01:59.230 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:01:59.230 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:01:59.231 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:01:59.231 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 174 M 2023-10-30T12:01:59.231 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 732 M 2023-10-30T12:01:59.231 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:01:59.886 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-radosgw-debuginfo-18.0.0-6839.gcc36dd85.el 14 MB/s | 174 MB 00:12 2023-10-30T12:01:59.887 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:01:59.887 INFO:teuthology.orchestra.run.smithi003.stdout:Total 14 MB/s | 174 MB 00:12 2023-10-30T12:01:59.888 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:01:59.892 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:01:59.892 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:02:01.002 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:02:01.003 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:02:11.216 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-radosgw-debuginfo-18.0.0-6839.gcc36dd85.el 15 MB/s | 174 MB 00:11 2023-10-30T12:02:11.217 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:02:11.217 INFO:teuthology.orchestra.run.smithi092.stdout:Total 15 MB/s | 174 MB 00:11 2023-10-30T12:02:11.217 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:02:11.221 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:02:11.221 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:02:12.276 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:02:12.276 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:02:19.860 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:02:19.952 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T12:02:24.449 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T12:02:24.719 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T12:02:24.720 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:02:24.720 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:02:24.720 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:02:24.720 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:02:24.720 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:02:24.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-test-debuginfo 2023-10-30T12:02:25.492 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:14 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:02:25.639 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:02:25.640 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:02:25.640 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:02:25.640 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:02:25.640 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:02:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 626 M 2023-10-30T12:02:25.641 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:02:25.641 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:02:25.641 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:02:25.641 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:02:25.642 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:02:25.642 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 626 M 2023-10-30T12:02:25.642 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 2.8 G 2023-10-30T12:02:25.642 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:02:30.237 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:02:30.403 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T12:02:35.025 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T12:02:35.300 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x 1/1 2023-10-30T12:02:35.300 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:02:35.300 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:02:35.300 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:02:35.300 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:02:35.301 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:02:35.483 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-test-debuginfo 2023-10-30T12:02:36.025 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:04:26 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:02:36.165 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:02:36.166 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:02:36.166 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:02:36.166 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:02:36.166 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:02:36.167 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 626 M 2023-10-30T12:02:36.167 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:02:36.167 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:02:36.167 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:02:36.167 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:02:36.168 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:02:36.168 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 626 M 2023-10-30T12:02:36.168 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 2.8 G 2023-10-30T12:02:36.168 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:03:28.473 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-test-debuginfo-18.0.0-6839.gcc36dd85.el9.x 12 MB/s | 626 MB 00:52 2023-10-30T12:03:28.473 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:03:28.474 INFO:teuthology.orchestra.run.smithi092.stdout:Total 12 MB/s | 626 MB 00:52 2023-10-30T12:03:28.474 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:03:28.479 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:03:28.479 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:03:32.201 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:03:32.201 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:03:41.568 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-test-debuginfo-18.0.0-6839.gcc36dd85.el9.x 8.2 MB/s | 626 MB 01:15 2023-10-30T12:03:41.569 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:03:41.570 INFO:teuthology.orchestra.run.smithi003.stdout:Total 8.2 MB/s | 626 MB 01:15 2023-10-30T12:03:41.570 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:03:41.575 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:03:41.576 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:03:45.494 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:03:45.494 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:04:37.615 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:04:37.724 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:04:46.795 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:04:47.054 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:04:47.054 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:04:47.054 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:04:47.055 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:47.055 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:04:47.055 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:04:47.285 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-base-debuginfo 2023-10-30T12:04:47.825 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:06:37 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:04:47.914 INFO:teuthology.orchestra.run.smithi092.stdout:Package ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:04:47.964 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:04:47.965 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-10-30T12:04:47.965 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:04:48.029 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-debuginfo 2023-10-30T12:04:48.569 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:06:38 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:04:48.709 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:04:48.736 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:04:48.736 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:04:48.737 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:04:48.737 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:04:48.737 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 19 M 2023-10-30T12:04:48.737 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:04:48.737 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:04:48.738 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:04:48.738 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:04:48.738 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:04:48.738 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 19 M 2023-10-30T12:04:48.739 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 101 M 2023-10-30T12:04:48.739 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:04:52.059 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-mgr-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 5.7 MB/s | 19 MB 00:03 2023-10-30T12:04:52.060 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:04:52.060 INFO:teuthology.orchestra.run.smithi092.stdout:Total 5.7 MB/s | 19 MB 00:03 2023-10-30T12:04:52.060 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:04:52.064 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:04:52.064 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:04:52.210 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:04:52.210 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:04:54.383 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:04:54.473 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:04:55.058 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:04:55.471 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:04:55.645 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:04:55.737 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:04:55.737 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:04:55.738 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:04:55.738 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:04:55.738 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:04:55.738 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:04:55.891 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mds-debuginfo 2023-10-30T12:04:56.435 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:06:46 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:04:56.575 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:04:56.575 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:04:56.576 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:04:56.576 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:04:56.576 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:04:56.576 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 47 M 2023-10-30T12:04:56.577 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:04:56.577 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:04:56.577 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:04:56.577 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:04:56.577 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:04:56.578 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 47 M 2023-10-30T12:04:56.578 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 170 M 2023-10-30T12:04:56.578 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:05:02.783 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:05:03.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-base-debuginfo 2023-10-30T12:05:03.908 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:06:52 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:05:04.001 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-base-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 is already installed. 2023-10-30T12:05:04.092 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:05:04.093 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-30T12:05:04.093 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:05:04.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-debuginfo 2023-10-30T12:05:04.710 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:06:53 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:05:04.809 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-mds-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 5.7 MB/s | 47 MB 00:08 2023-10-30T12:05:04.810 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:04.811 INFO:teuthology.orchestra.run.smithi092.stdout:Total 5.7 MB/s | 47 MB 00:08 2023-10-30T12:05:04.811 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:05:04.815 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:05:04.815 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:05:04.856 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:05:04.857 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:05:04.857 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:05:04.857 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:05:04.857 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:05:04.858 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 19 M 2023-10-30T12:05:04.858 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:04.858 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:05:04.858 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:05:04.858 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:05:04.859 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:04.859 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 19 M 2023-10-30T12:05:04.859 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 101 M 2023-10-30T12:05:04.859 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:05:05.127 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:05:05.127 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:05:07.128 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-mgr-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 8.5 MB/s | 19 MB 00:02 2023-10-30T12:05:07.129 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:07.129 INFO:teuthology.orchestra.run.smithi003.stdout:Total 8.5 MB/s | 19 MB 00:02 2023-10-30T12:05:07.130 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:05:07.134 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:05:07.134 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:05:07.290 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:05:07.290 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:05:09.616 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:05:09.713 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:05:09.781 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:05:09.890 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:05:10.707 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:05:10.974 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:05:10.974 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:10.975 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:05:10.975 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:10.975 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:10.975 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:05:11.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mds-debuginfo 2023-10-30T12:05:11.196 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:05:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:05:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:05:11.455 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:05:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:05:11.456 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:05:11.608 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mon-debuginfo 2023-10-30T12:05:11.733 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:07:00 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:05:11.880 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:05:11.881 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:05:11.882 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:05:11.882 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:05:11.882 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:05:11.882 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 47 M 2023-10-30T12:05:11.883 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:11.883 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:05:11.883 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:05:11.883 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:05:11.884 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:11.884 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 47 M 2023-10-30T12:05:11.884 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 170 M 2023-10-30T12:05:11.884 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:05:12.145 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:07:02 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:05:12.285 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:05:12.286 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:05:12.286 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:05:12.286 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:05:12.287 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:05:12.287 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 112 M 2023-10-30T12:05:12.287 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:05:12.287 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:05:12.287 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:05:12.287 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:05:12.288 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:05:12.288 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 112 M 2023-10-30T12:05:12.288 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 509 M 2023-10-30T12:05:12.288 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:05:15.188 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-mds-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 14 MB/s | 47 MB 00:03 2023-10-30T12:05:15.189 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:15.190 INFO:teuthology.orchestra.run.smithi003.stdout:Total 14 MB/s | 47 MB 00:03 2023-10-30T12:05:15.190 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:05:15.194 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:05:15.194 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:05:15.522 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:05:15.522 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:05:20.467 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:05:20.556 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:05:21.933 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:05:22.200 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:05:22.200 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:22.200 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:05:22.201 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:22.201 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:22.201 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:05:22.368 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-mon-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 11 MB/s | 112 MB 00:10 2023-10-30T12:05:22.369 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:22.369 INFO:teuthology.orchestra.run.smithi092.stdout:Total 11 MB/s | 112 MB 00:10 2023-10-30T12:05:22.370 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:05:22.373 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:05:22.374 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:05:22.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mon-debuginfo 2023-10-30T12:05:22.970 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:07:11 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:05:23.064 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:05:23.064 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:05:23.117 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:05:23.117 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:05:23.117 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:05:23.118 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:05:23.118 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:05:23.118 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 112 M 2023-10-30T12:05:23.118 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:23.119 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:05:23.119 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:05:23.119 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:05:23.119 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:23.120 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 112 M 2023-10-30T12:05:23.120 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 509 M 2023-10-30T12:05:23.120 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:05:30.131 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-mon-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 16 MB/s | 112 MB 00:07 2023-10-30T12:05:30.132 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:05:30.133 INFO:teuthology.orchestra.run.smithi003.stdout:Total 16 MB/s | 112 MB 00:07 2023-10-30T12:05:30.133 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:05:30.137 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:05:30.137 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:05:30.866 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:05:30.867 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:05:34.725 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:05:34.906 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:05:38.134 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:05:38.417 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:05:38.417 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:05:38.417 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:05:38.418 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:38.418 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:05:38.418 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:05:38.567 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-osd-debuginfo 2023-10-30T12:05:39.108 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:07:29 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:05:39.246 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:05:39.247 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:05:39.247 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:05:39.247 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:05:39.247 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:05:39.248 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 315 M 2023-10-30T12:05:39.248 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:05:39.248 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:05:39.248 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:05:39.248 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:05:39.249 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:05:39.249 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 315 M 2023-10-30T12:05:39.249 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 1.4 G 2023-10-30T12:05:39.249 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:05:43.303 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:05:43.407 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:05:46.678 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:05:46.962 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:05:46.962 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:46.963 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:05:46.963 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:05:46.963 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:46.963 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:05:47.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-osd-debuginfo 2023-10-30T12:05:47.718 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:07:36 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:05:47.865 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:05:47.866 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:05:47.866 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:05:47.866 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:05:47.867 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:05:47.867 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 315 M 2023-10-30T12:05:47.867 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:47.867 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:05:47.867 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:05:47.868 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:05:47.868 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:05:47.868 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 315 M 2023-10-30T12:05:47.868 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 1.4 G 2023-10-30T12:05:47.868 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:06:17.555 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-osd-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 11 MB/s | 315 MB 00:29 2023-10-30T12:06:17.555 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:06:17.556 INFO:teuthology.orchestra.run.smithi003.stdout:Total 11 MB/s | 315 MB 00:29 2023-10-30T12:06:17.556 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:06:17.560 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:06:17.560 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:06:19.540 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:06:19.541 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:06:26.251 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-osd-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 6.7 MB/s | 315 MB 00:47 2023-10-30T12:06:26.251 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:06:26.252 INFO:teuthology.orchestra.run.smithi092.stdout:Total 6.7 MB/s | 315 MB 00:47 2023-10-30T12:06:26.252 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:06:26.256 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:06:26.256 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:06:28.135 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:06:28.135 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:06:53.755 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:06:54.328 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:07:00.385 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:07:00.552 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:07:01.730 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:07:02.041 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:07:02.041 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:02.041 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:07:02.042 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:02.042 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:02.042 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:07:02.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-fuse-debuginfo 2023-10-30T12:07:02.835 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:08:51 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:07:02.979 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:07:02.980 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:02.980 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:07:02.980 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:02.981 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:07:02.981 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.8 M 2023-10-30T12:07:02.981 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:02.981 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:07:02.981 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:02.982 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:07:02.982 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:02.982 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 5.8 M 2023-10-30T12:07:02.982 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 28 M 2023-10-30T12:07:02.982 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:07:03.520 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-fuse-debuginfo-18.0.0-6839.gcc36dd85.el9.x 11 MB/s | 5.8 MB 00:00 2023-10-30T12:07:03.521 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:03.521 INFO:teuthology.orchestra.run.smithi003.stdout:Total 11 MB/s | 5.8 MB 00:00 2023-10-30T12:07:03.521 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:07:03.525 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:07:03.525 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:07:03.597 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:07:03.598 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:07:04.346 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:07:04.479 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:07:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:07:05.367 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:07:05.367 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:05.368 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:07:05.368 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:05.368 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:05.368 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:07:05.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados-devel-debuginfo 2023-10-30T12:07:06.132 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:08:55 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:07:06.279 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:07:06.281 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:06.281 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:07:06.281 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:06.282 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:07:06.282 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 108 k 2023-10-30T12:07:06.282 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:06.283 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:07:06.283 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:06.284 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:07:06.284 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:06.284 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 108 k 2023-10-30T12:07:06.285 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 440 k 2023-10-30T12:07:06.285 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:07:06.631 INFO:teuthology.orchestra.run.smithi003.stdout:librados-devel-debuginfo-18.0.0-6839.gcc36dd85. 310 kB/s | 108 kB 00:00 2023-10-30T12:07:06.631 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:06.632 INFO:teuthology.orchestra.run.smithi003.stdout:Total 309 kB/s | 108 kB 00:00 2023-10-30T12:07:06.632 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:07:06.635 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:07:06.635 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:07:06.671 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:07:06.671 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:07:06.789 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:07:06.898 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:07:07.311 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:07:07.593 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:07:07.593 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:07.594 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:07:07.594 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:07.594 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:07.594 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:07:07.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs2-debuginfo 2023-10-30T12:07:08.385 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:08:57 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:07:08.531 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:07:08.532 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:08.532 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:07:08.532 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:08.533 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:07:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.1 M 2023-10-30T12:07:08.533 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:08.533 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:07:08.533 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:08.534 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:07:08.534 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:08.534 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 5.1 M 2023-10-30T12:07:08.534 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 24 M 2023-10-30T12:07:08.534 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:07:08.671 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:07:08.956 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:07:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:08.957 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:07:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:08.958 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:07:09.042 INFO:teuthology.orchestra.run.smithi003.stdout:libcephfs2-debuginfo-18.0.0-6839.gcc36dd85.el9. 10 MB/s | 5.1 MB 00:00 2023-10-30T12:07:09.042 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:09.043 INFO:teuthology.orchestra.run.smithi003.stdout:Total 10 MB/s | 5.1 MB 00:00 2023-10-30T12:07:09.043 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:07:09.047 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:07:09.047 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:07:09.116 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:07:09.117 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:07:09.132 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-fuse-debuginfo 2023-10-30T12:07:09.677 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:08:59 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:07:09.798 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:07:09.816 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:07:09.818 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:09.818 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:07:09.818 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:09.818 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:07:09.818 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.8 M 2023-10-30T12:07:09.819 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:09.819 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:07:09.819 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:09.819 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:07:09.819 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:09.820 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 5.8 M 2023-10-30T12:07:09.820 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 28 M 2023-10-30T12:07:09.820 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:07:09.893 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:07:10.313 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-fuse-debuginfo-18.0.0-6839.gcc36dd85.el9.x 12 MB/s | 5.8 MB 00:00 2023-10-30T12:07:10.315 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:10.315 INFO:teuthology.orchestra.run.smithi092.stdout:Total 12 MB/s | 5.8 MB 00:00 2023-10-30T12:07:10.315 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:07:10.318 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:07:10.318 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:07:10.385 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:07:10.385 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:07:10.445 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:07:10.748 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:07:10.749 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:10.749 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:07:10.749 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:10.749 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:10.749 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:07:10.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados2-debuginfo 2023-10-30T12:07:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:07:11.228 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:07:11.518 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:09:00 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:07:11.662 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:07:11.663 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:11.663 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:07:11.664 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:11.664 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:07:11.664 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 45 M 2023-10-30T12:07:11.664 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:11.664 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:07:11.665 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:11.665 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:07:11.665 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:11.665 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 45 M 2023-10-30T12:07:11.665 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 255 M 2023-10-30T12:07:11.666 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:07:11.784 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:07:12.046 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:07:12.046 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:12.046 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:07:12.047 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:12.047 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:12.047 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:07:12.192 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librados-devel-debuginfo 2023-10-30T12:07:12.732 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:09:02 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:07:12.869 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:07:12.870 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:12.871 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:07:12.871 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:12.871 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:07:12.871 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 108 k 2023-10-30T12:07:12.871 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:12.872 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:07:12.872 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:12.872 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:07:12.872 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:12.872 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 108 k 2023-10-30T12:07:12.873 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 440 k 2023-10-30T12:07:12.873 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:07:13.249 INFO:teuthology.orchestra.run.smithi092.stdout:librados-devel-debuginfo-18.0.0-6839.gcc36dd85. 287 kB/s | 108 kB 00:00 2023-10-30T12:07:13.250 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:13.250 INFO:teuthology.orchestra.run.smithi092.stdout:Total 287 kB/s | 108 kB 00:00 2023-10-30T12:07:13.250 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:07:13.253 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:07:13.253 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:07:13.288 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:07:13.288 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:07:13.405 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:07:13.513 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:07:13.872 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:07:14.141 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:07:14.141 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:14.141 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:07:14.141 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:14.142 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:14.142 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:07:14.306 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install libcephfs2-debuginfo 2023-10-30T12:07:14.428 INFO:teuthology.orchestra.run.smithi003.stdout:librados2-debuginfo-18.0.0-6839.gcc36dd85.el9.x 16 MB/s | 45 MB 00:02 2023-10-30T12:07:14.428 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:14.429 INFO:teuthology.orchestra.run.smithi003.stdout:Total 16 MB/s | 45 MB 00:02 2023-10-30T12:07:14.429 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:07:14.434 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:07:14.434 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:07:14.760 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:07:14.760 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:07:14.848 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:09:04 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:07:14.988 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:07:14.988 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:14.989 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:07:14.989 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:14.989 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:07:14.989 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 5.1 M 2023-10-30T12:07:14.990 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:14.990 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:07:14.990 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:14.990 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:07:14.990 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:14.991 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 5.1 M 2023-10-30T12:07:14.991 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 24 M 2023-10-30T12:07:14.991 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:07:15.643 INFO:teuthology.orchestra.run.smithi092.stdout:libcephfs2-debuginfo-18.0.0-6839.gcc36dd85.el9. 7.9 MB/s | 5.1 MB 00:00 2023-10-30T12:07:15.644 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:15.645 INFO:teuthology.orchestra.run.smithi092.stdout:Total 7.8 MB/s | 5.1 MB 00:00 2023-10-30T12:07:15.645 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:07:15.648 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:07:15.648 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:07:15.713 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:07:15.713 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:07:16.359 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:07:16.450 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:07:16.984 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:07:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:07:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:17.257 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:07:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:17.258 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:07:17.425 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librados2-debuginfo 2023-10-30T12:07:17.999 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:09:07 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:07:18.137 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:07:18.138 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:07:18.139 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:18.139 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:07:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 45 M 2023-10-30T12:07:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:18.140 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:07:18.140 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:18.140 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:07:18.140 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:18.140 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 45 M 2023-10-30T12:07:18.141 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 255 M 2023-10-30T12:07:18.141 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:07:20.198 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:07:20.304 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:07:21.720 INFO:teuthology.orchestra.run.smithi092.stdout:librados2-debuginfo-18.0.0-6839.gcc36dd85.el9.x 13 MB/s | 45 MB 00:03 2023-10-30T12:07:21.721 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:21.722 INFO:teuthology.orchestra.run.smithi092.stdout:Total 13 MB/s | 45 MB 00:03 2023-10-30T12:07:21.722 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:07:21.726 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:07:21.726 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:07:22.033 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:07:22.034 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:07:22.224 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:07:22.506 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:07:22.506 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:22.506 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:07:22.507 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:22.507 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:22.507 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:07:22.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librbd1-debuginfo 2023-10-30T12:07:23.276 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:09:12 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:07:23.421 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:07:23.422 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:23.422 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:07:23.422 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:23.422 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:07:23.423 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 51 M 2023-10-30T12:07:23.423 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:23.423 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:07:23.423 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:23.423 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:07:23.424 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:23.424 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 51 M 2023-10-30T12:07:23.424 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 298 M 2023-10-30T12:07:23.424 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:07:26.046 INFO:teuthology.orchestra.run.smithi003.stdout:librbd1-debuginfo-18.0.0-6839.gcc36dd85.el9.x86 19 MB/s | 51 MB 00:02 2023-10-30T12:07:26.047 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:26.048 INFO:teuthology.orchestra.run.smithi003.stdout:Total 19 MB/s | 51 MB 00:02 2023-10-30T12:07:26.048 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:07:26.052 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:07:26.053 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:07:26.412 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:07:26.412 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:07:27.140 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:07:27.225 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:07:29.037 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:07:29.311 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_ 1/1 2023-10-30T12:07:29.311 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:29.311 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:07:29.311 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:29.312 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:29.312 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:07:29.477 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librbd1-debuginfo 2023-10-30T12:07:30.024 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:09:20 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:07:30.164 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:07:30.165 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:30.165 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:07:30.166 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:30.166 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:07:30.166 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 51 M 2023-10-30T12:07:30.166 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:30.167 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:07:30.167 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:30.167 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:07:30.167 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:30.167 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 51 M 2023-10-30T12:07:30.168 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 298 M 2023-10-30T12:07:30.168 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:07:32.616 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:07:32.730 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:07:34.934 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:07:35.220 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:07:35.220 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:35.220 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:07:35.221 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:35.221 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:35.221 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:07:35.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-cephfs-debuginfo 2023-10-30T12:07:35.999 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:09:24 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:07:36.144 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:07:36.145 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:36.146 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:07:36.146 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:36.146 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:07:36.146 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 615 k 2023-10-30T12:07:36.146 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:36.147 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:07:36.147 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:36.147 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:07:36.147 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:36.147 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 615 k 2023-10-30T12:07:36.148 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 2.1 M 2023-10-30T12:07:36.148 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:07:37.124 INFO:teuthology.orchestra.run.smithi003.stdout:python3-cephfs-debuginfo-18.0.0-6839.gcc36dd85. 629 kB/s | 615 kB 00:00 2023-10-30T12:07:37.125 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:37.126 INFO:teuthology.orchestra.run.smithi003.stdout:Total 628 kB/s | 615 kB 00:00 2023-10-30T12:07:37.126 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:07:37.129 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:07:37.129 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:07:37.171 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:07:37.172 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:07:37.348 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:07:37.459 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:07:37.902 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:07:38.194 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:07:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:38.195 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:07:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:38.196 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:07:38.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rados-debuginfo 2023-10-30T12:07:38.948 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:09:27 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:07:39.093 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:07:39.095 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:39.095 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:07:39.096 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:39.096 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:07:39.096 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 770 k 2023-10-30T12:07:39.097 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:39.097 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:07:39.097 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:39.097 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:07:39.098 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:39.098 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 770 k 2023-10-30T12:07:39.098 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 2.7 M 2023-10-30T12:07:39.099 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:07:39.993 INFO:teuthology.orchestra.run.smithi003.stdout:python3-rados-debuginfo-18.0.0-6839.gcc36dd85.e 858 kB/s | 770 kB 00:00 2023-10-30T12:07:39.994 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:39.994 INFO:teuthology.orchestra.run.smithi003.stdout:Total 856 kB/s | 770 kB 00:00 2023-10-30T12:07:39.995 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:07:39.998 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:07:39.998 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:07:40.044 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:07:40.045 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:07:40.250 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:07:40.354 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T12:07:40.769 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T12:07:41.044 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T12:07:41.044 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:41.045 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:07:41.045 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:41.045 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:41.045 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:07:41.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rbd-debuginfo 2023-10-30T12:07:41.728 INFO:teuthology.orchestra.run.smithi092.stdout:librbd1-debuginfo-18.0.0-6839.gcc36dd85.el9.x86 4.4 MB/s | 51 MB 00:11 2023-10-30T12:07:41.729 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:41.730 INFO:teuthology.orchestra.run.smithi092.stdout:Total 4.4 MB/s | 51 MB 00:11 2023-10-30T12:07:41.730 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:07:41.734 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:07:41.734 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:07:41.788 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:09:30 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:07:41.935 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:07:41.935 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:41.935 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:07:41.935 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:41.936 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:07:41.936 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.1 M 2023-10-30T12:07:41.936 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:41.936 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:07:41.936 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:41.937 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:07:41.937 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:41.937 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 1.1 M 2023-10-30T12:07:41.938 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 4.3 M 2023-10-30T12:07:41.938 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:07:42.074 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:07:42.075 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:07:42.896 INFO:teuthology.orchestra.run.smithi003.stdout:python3-rbd-debuginfo-18.0.0-6839.gcc36dd85.el9 1.2 MB/s | 1.1 MB 00:00 2023-10-30T12:07:42.897 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:42.897 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.2 MB/s | 1.1 MB 00:00 2023-10-30T12:07:42.898 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:07:42.902 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:07:42.902 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:07:42.950 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:07:42.951 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:07:43.180 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:07:43.289 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:07:43.714 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:07:44.001 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:07:44.001 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:44.001 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:07:44.002 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:44.002 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:44.002 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:07:44.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rgw-debuginfo 2023-10-30T12:07:44.773 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:09:33 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:07:44.918 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:07:44.919 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:44.919 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:07:44.919 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:44.919 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:07:44.920 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 337 k 2023-10-30T12:07:44.920 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:44.920 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:07:44.920 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:44.920 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:07:44.921 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:44.921 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 337 k 2023-10-30T12:07:44.921 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 1.0 M 2023-10-30T12:07:44.921 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:07:45.524 INFO:teuthology.orchestra.run.smithi003.stdout:python3-rgw-debuginfo-18.0.0-6839.gcc36dd85.el9 557 kB/s | 337 kB 00:00 2023-10-30T12:07:45.525 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:45.525 INFO:teuthology.orchestra.run.smithi003.stdout:Total 556 kB/s | 337 kB 00:00 2023-10-30T12:07:45.525 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:07:45.529 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:07:45.529 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:07:45.570 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:07:45.570 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:07:45.710 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:07:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:07:46.329 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:07:46.602 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:07:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:46.603 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:07:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:46.604 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:07:46.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-fuse-debuginfo 2023-10-30T12:07:47.357 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:09:36 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:07:47.503 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:07:47.504 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:47.504 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:07:47.504 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:47.504 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:07:47.505 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 506 k 2023-10-30T12:07:47.505 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:47.505 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:07:47.505 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:47.505 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:07:47.506 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:47.506 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 506 k 2023-10-30T12:07:47.506 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 2.5 M 2023-10-30T12:07:47.506 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:07:47.956 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:07:48.051 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:07:48.502 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-fuse-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 507 kB/s | 506 kB 00:00 2023-10-30T12:07:48.503 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:48.503 INFO:teuthology.orchestra.run.smithi003.stdout:Total 506 kB/s | 506 kB 00:00 2023-10-30T12:07:48.504 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:07:48.507 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:07:48.508 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:07:48.553 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:07:48.553 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:07:48.745 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:07:48.854 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:07:49.255 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:07:49.526 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:07:49.526 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:49.527 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:07:49.527 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:49.527 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:49.527 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:07:49.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-mirror-debuginfo 2023-10-30T12:07:50.191 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:07:50.279 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:09:39 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:07:50.425 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:07:50.426 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:50.426 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:07:50.427 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:50.427 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:07:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 51 M 2023-10-30T12:07:50.427 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:50.428 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:07:50.428 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:07:50.428 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:07:50.428 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:07:50.428 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 51 M 2023-10-30T12:07:50.429 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 289 M 2023-10-30T12:07:50.429 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:07:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:07:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:50.460 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:07:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:50.461 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:50.461 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:07:50.631 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-cephfs-debuginfo 2023-10-30T12:07:51.176 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:09:41 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:07:51.316 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:07:51.317 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:51.317 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:07:51.317 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:51.318 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:07:51.318 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 615 k 2023-10-30T12:07:51.318 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:51.318 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:07:51.319 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:51.319 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:07:51.319 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:51.319 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 615 k 2023-10-30T12:07:51.319 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 2.1 M 2023-10-30T12:07:51.320 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:07:52.297 INFO:teuthology.orchestra.run.smithi092.stdout:python3-cephfs-debuginfo-18.0.0-6839.gcc36dd85. 628 kB/s | 615 kB 00:00 2023-10-30T12:07:52.297 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:52.298 INFO:teuthology.orchestra.run.smithi092.stdout:Total 627 kB/s | 615 kB 00:00 2023-10-30T12:07:52.298 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:07:52.301 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:07:52.301 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:07:52.341 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:07:52.342 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:07:52.505 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:07:52.611 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:07:53.024 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:07:53.297 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9 1/1 2023-10-30T12:07:53.297 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:53.298 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:07:53.298 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:53.298 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:53.298 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:07:53.444 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rados-debuginfo 2023-10-30T12:07:53.985 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:09:43 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:07:54.125 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:07:54.126 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:07:54.127 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:54.127 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:07:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 770 k 2023-10-30T12:07:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:54.128 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:07:54.128 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:54.128 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:07:54.128 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:54.128 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 770 k 2023-10-30T12:07:54.129 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 2.7 M 2023-10-30T12:07:54.129 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:07:55.049 INFO:teuthology.orchestra.run.smithi092.stdout:python3-rados-debuginfo-18.0.0-6839.gcc36dd85.e 835 kB/s | 770 kB 00:00 2023-10-30T12:07:55.050 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:55.050 INFO:teuthology.orchestra.run.smithi092.stdout:Total 834 kB/s | 770 kB 00:00 2023-10-30T12:07:55.051 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:07:55.054 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:07:55.054 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:07:55.094 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:07:55.095 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:07:55.280 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:07:55.381 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T12:07:55.771 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T12:07:56.038 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9. 1/1 2023-10-30T12:07:56.039 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:56.039 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:07:56.039 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:56.039 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:56.039 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:07:56.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rbd-debuginfo 2023-10-30T12:07:56.742 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:09:46 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:07:56.880 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:07:56.881 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:56.882 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:07:56.882 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:56.882 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:07:56.882 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.1 M 2023-10-30T12:07:56.883 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:56.883 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:07:56.883 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:56.883 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:07:56.883 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:56.884 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 1.1 M 2023-10-30T12:07:56.884 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 4.3 M 2023-10-30T12:07:56.884 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:07:57.835 INFO:teuthology.orchestra.run.smithi092.stdout:python3-rbd-debuginfo-18.0.0-6839.gcc36dd85.el9 1.2 MB/s | 1.1 MB 00:00 2023-10-30T12:07:57.836 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:07:57.836 INFO:teuthology.orchestra.run.smithi092.stdout:Total 1.2 MB/s | 1.1 MB 00:00 2023-10-30T12:07:57.836 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:07:57.840 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:07:57.840 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:07:57.882 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:07:57.883 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:07:58.098 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:07:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:07:58.584 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:07:58.853 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:07:58.854 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:58.854 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:07:58.854 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:07:58.854 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:58.855 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:07:59.044 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rgw-debuginfo 2023-10-30T12:07:59.590 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:09:49 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:07:59.728 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:07:59.729 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:59.730 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:07:59.730 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:59.730 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:07:59.730 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 337 k 2023-10-30T12:07:59.731 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:59.731 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:07:59.731 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:07:59.731 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:07:59.731 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:07:59.732 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 337 k 2023-10-30T12:07:59.732 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 1.0 M 2023-10-30T12:07:59.732 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:08:00.438 INFO:teuthology.orchestra.run.smithi092.stdout:python3-rgw-debuginfo-18.0.0-6839.gcc36dd85.el9 476 kB/s | 337 kB 00:00 2023-10-30T12:08:00.439 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:08:00.439 INFO:teuthology.orchestra.run.smithi092.stdout:Total 475 kB/s | 337 kB 00:00 2023-10-30T12:08:00.440 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:08:00.443 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:08:00.443 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:08:00.480 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:08:00.481 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:08:00.617 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:08:00.720 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:08:01.097 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:08:01.369 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x8 1/1 2023-10-30T12:08:01.369 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:01.370 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:08:01.370 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:08:01.370 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:01.370 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:08:01.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-fuse-debuginfo 2023-10-30T12:08:02.070 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:09:52 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:08:02.209 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:08:02.210 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:08:02.210 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:08:02.210 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:08:02.210 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:08:02.211 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 506 k 2023-10-30T12:08:02.211 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:02.211 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:08:02.211 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:08:02.212 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:08:02.212 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:02.212 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 506 k 2023-10-30T12:08:02.212 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 2.5 M 2023-10-30T12:08:02.212 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:08:02.933 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-fuse-debuginfo-18.0.0-6839.gcc36dd85.el9.x8 700 kB/s | 506 kB 00:00 2023-10-30T12:08:02.933 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:08:02.934 INFO:teuthology.orchestra.run.smithi092.stdout:Total 699 kB/s | 506 kB 00:00 2023-10-30T12:08:02.934 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:08:02.937 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:08:02.937 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:08:02.974 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:08:02.975 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:08:03.130 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:08:03.223 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:08:03.591 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:08:03.853 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_6 1/1 2023-10-30T12:08:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:03.854 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:08:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:08:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:03.855 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:08:04.006 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-mirror-debuginfo 2023-10-30T12:08:04.548 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:09:54 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:08:04.686 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:08:04.687 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:08:04.688 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:08:04.688 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:08:04.688 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:08:04.688 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 51 M 2023-10-30T12:08:04.688 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:04.689 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:08:04.689 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:08:04.689 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:08:04.689 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:04.689 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 51 M 2023-10-30T12:08:04.690 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 289 M 2023-10-30T12:08:04.690 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:08:08.628 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-mirror-debuginfo-18.0.0-6839.gcc36dd85.el9. 2.8 MB/s | 51 MB 00:18 2023-10-30T12:08:08.628 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:08:08.629 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.8 MB/s | 51 MB 00:18 2023-10-30T12:08:08.629 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:08:08.633 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:08:08.634 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:08:08.986 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:08:08.986 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:08:11.809 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-mirror-debuginfo-18.0.0-6839.gcc36dd85.el9. 7.1 MB/s | 51 MB 00:07 2023-10-30T12:08:11.810 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:08:11.810 INFO:teuthology.orchestra.run.smithi092.stdout:Total 7.1 MB/s | 51 MB 00:07 2023-10-30T12:08:11.810 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:08:11.814 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:08:11.815 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:08:12.149 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:08:12.150 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:08:15.125 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:08:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:08:17.319 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:08:17.614 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:08:17.615 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:17.615 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:08:17.615 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:08:17.615 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:17.615 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:08:17.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-nbd-debuginfo 2023-10-30T12:08:17.951 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:08:18.048 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:08:18.382 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:10:07 ago on Mon 30 Oct 2023 11:58:11 AM UTC. 2023-10-30T12:08:18.527 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-30T12:08:18.528 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:08:18.529 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-30T12:08:18.529 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:08:18.529 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-30T12:08:18.529 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.2 M 2023-10-30T12:08:18.530 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:18.530 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-30T12:08:18.530 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-30T12:08:18.530 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-30T12:08:18.530 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:18.531 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 1.2 M 2023-10-30T12:08:18.531 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 5.4 M 2023-10-30T12:08:18.531 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-30T12:08:19.996 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-nbd-debuginfo-18.0.0-6839.gcc36dd85.el9.x86 859 kB/s | 1.2 MB 00:01 2023-10-30T12:08:19.997 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:08:19.997 INFO:teuthology.orchestra.run.smithi003.stdout:Total 858 kB/s | 1.2 MB 00:01 2023-10-30T12:08:19.998 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-30T12:08:20.001 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-30T12:08:20.001 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-30T12:08:20.050 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-30T12:08:20.050 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-30T12:08:20.069 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:08:20.302 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-30T12:08:20.339 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86 1/1 2023-10-30T12:08:20.340 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:20.340 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:08:20.340 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:08:20.341 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:20.341 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:08:20.406 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:08:20.503 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-nbd-debuginfo 2023-10-30T12:08:20.852 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:08:21.044 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:10:11 ago on Mon 30 Oct 2023 11:58:10 AM UTC. 2023-10-30T12:08:21.112 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:08:21.113 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:21.113 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-30T12:08:21.113 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:08:21.113 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:21.113 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-30T12:08:21.183 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-10-30T12:08:21.184 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:08:21.184 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-10-30T12:08:21.184 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:08:21.184 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-10-30T12:08:21.184 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-6839.gcc36dd85.el9 ceph 1.2 M 2023-10-30T12:08:21.185 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:21.185 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-10-30T12:08:21.185 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-10-30T12:08:21.185 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-10-30T12:08:21.185 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:21.186 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 1.2 M 2023-10-30T12:08:21.186 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 5.4 M 2023-10-30T12:08:21.186 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-10-30T12:08:21.311 DEBUG:teuthology.parallel:result is None 2023-10-30T12:08:22.318 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-nbd-debuginfo-18.0.0-6839.gcc36dd85.el9.x86 1.1 MB/s | 1.2 MB 00:01 2023-10-30T12:08:22.319 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:08:22.319 INFO:teuthology.orchestra.run.smithi092.stdout:Total 1.1 MB/s | 1.2 MB 00:01 2023-10-30T12:08:22.319 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-10-30T12:08:22.322 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-10-30T12:08:22.323 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-10-30T12:08:22.363 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-10-30T12:08:22.363 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-10-30T12:08:22.592 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-10-30T12:08:22.701 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:08:23.112 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:08:23.389 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 1/1 2023-10-30T12:08:23.390 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:23.390 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-10-30T12:08:23.390 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-debuginfo-2:18.0.0-6839.gcc36dd85.el9.x86_64 2023-10-30T12:08:23.390 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:23.390 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-10-30T12:08:23.551 DEBUG:teuthology.parallel:result is None 2023-10-30T12:08:23.551 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cc36dd85d142c6fdb77515cad8af108cec98a86d 2023-10-30T12:08:23.701 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-30T12:08:23.726 INFO:teuthology.orchestra.run.smithi003.stdout:18.0.0-6839.gcc36dd85.el9 2023-10-30T12:08:23.727 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6839.gcc36dd85.el9 2023-10-30T12:08:23.727 INFO:teuthology.task.install:The correct ceph version 18.0.0-6839.gcc36dd85 is installed. 2023-10-30T12:08:23.728 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cc36dd85d142c6fdb77515cad8af108cec98a86d 2023-10-30T12:08:23.949 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-30T12:08:23.987 INFO:teuthology.orchestra.run.smithi092.stdout:18.0.0-6839.gcc36dd85.el9 2023-10-30T12:08:23.988 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6839.gcc36dd85.el9 2023-10-30T12:08:23.988 INFO:teuthology.task.install:The correct ceph version 18.0.0-6839.gcc36dd85 is installed. 2023-10-30T12:08:23.989 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-30T12:08:23.989 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:23.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T12:08:24.026 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:08:24.027 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T12:08:24.054 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-30T12:08:24.054 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:24.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T12:08:24.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T12:08:24.176 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:08:24.176 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T12:08:24.203 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T12:08:24.269 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-30T12:08:24.270 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:24.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T12:08:24.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T12:08:24.377 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:08:24.377 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T12:08:24.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T12:08:24.471 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-30T12:08:24.471 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:24.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T12:08:24.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T12:08:24.577 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:08:24.578 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T12:08:24.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T12:08:24.671 INFO:teuthology.run_tasks:Running task ceph... 2023-10-30T12:08:24.737 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-30T12:08:24.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /var/log/ceph 2023-10-30T12:08:24.741 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /var/log/ceph 2023-10-30T12:08:24.770 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-30T12:08:24.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T12:08:24.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T12:08:24.842 INFO:tasks.ceph:Creating extra log directories... 2023-10-30T12:08:24.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T12:08:24.885 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T12:08:24.923 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-30T12:08:24.924 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-30T12:08:24.924 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7441070', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': 'd0db9f2f95b9d6344950185071cace2a50d296eb'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 1876, 'sha1': 'cc36dd85d142c6fdb77515cad8af108cec98a86d', 'sleep_before_teardown': 0, 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd0db9f2f95b9d6344950185071cace2a50d296eb', 'targets': {'smithi003.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB0aWaga+dsDrTwqHzQWFfUjBQaami3eA1PWTEDHnqNfmchWzCWJ7rBcGnS6wHE8cDcDGGiFpluElQlxZf8rN+w=', 'smithi092.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBABPeEmKQ4kin+AOWQHfOC8gZ7lIz1qIaXii2x99f4NDEOM4MA9xORRfqdfj44euxLjUEZy/r5gm4sjfbgrWH7E='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-30_10:36:20', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-10-30T12:08:24.925 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T12:08:24.957 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T12:08:24.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T12:08:25.013 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T12:08:25.041 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:25.041 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T12:08:25.097 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T12:08:25.097 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2023-10-30T12:08:25.152 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T12:08:25.153 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:08:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-10-30T12:08:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:08:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:08:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-10-30 11:59:56.280978718 +0000 2023-10-30T12:08:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-10-30 11:57:23.106543000 +0000 2023-10-30T12:08:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-10-30 11:57:23.106543000 +0000 2023-10-30T12:08:25.154 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2023-10-30 11:57:23.106543000 +0000 2023-10-30T12:08:25.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T12:08:25.221 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-10-30T12:08:25.221 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-10-30T12:08:25.222 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000189407 s, 2.7 MB/s 2023-10-30T12:08:25.223 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T12:08:25.279 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2023-10-30T12:08:25.334 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T12:08:25.334 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:08:25.335 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-10-30T12:08:25.335 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:08:25.335 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:08:25.335 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-10-30 11:59:56.280978718 +0000 2023-10-30T12:08:25.336 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-10-30 11:57:23.627536407 +0000 2023-10-30T12:08:25.336 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-10-30 11:57:23.627536407 +0000 2023-10-30T12:08:25.336 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2023-10-30 11:57:23.627536407 +0000 2023-10-30T12:08:25.336 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T12:08:25.405 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-10-30T12:08:25.406 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-10-30T12:08:25.406 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000222958 s, 2.3 MB/s 2023-10-30T12:08:25.407 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T12:08:25.463 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2023-10-30T12:08:25.518 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T12:08:25.519 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:08:25.519 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-10-30T12:08:25.519 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:08:25.519 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:08:25.519 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-10-30 11:59:56.280978718 +0000 2023-10-30T12:08:25.520 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-10-30 11:57:24.120530168 +0000 2023-10-30T12:08:25.520 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-10-30 11:57:24.120530168 +0000 2023-10-30T12:08:25.520 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2023-10-30 11:57:24.120530168 +0000 2023-10-30T12:08:25.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T12:08:25.587 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-10-30T12:08:25.587 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-10-30T12:08:25.588 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000182951 s, 2.8 MB/s 2023-10-30T12:08:25.589 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T12:08:25.645 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2023-10-30T12:08:25.700 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T12:08:25.700 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:08:25.700 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-10-30T12:08:25.701 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:08:25.701 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:08:25.701 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-10-30 11:59:56.281978705 +0000 2023-10-30T12:08:25.701 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-10-30 11:57:24.638523612 +0000 2023-10-30T12:08:25.701 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-10-30 11:57:24.638523612 +0000 2023-10-30T12:08:25.702 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2023-10-30 11:57:24.638523612 +0000 2023-10-30T12:08:25.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T12:08:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-10-30T12:08:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-10-30T12:08:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000242914 s, 2.1 MB/s 2023-10-30T12:08:25.770 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T12:08:25.827 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:08:25.828 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:08:25.828 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T12:08:25.845 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T12:08:25.845 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 2023-10-30T12:08:25.901 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T12:08:25.901 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:08:25.901 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-10-30T12:08:25.902 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:08:25.902 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:08:25.902 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-10-30 11:59:58.279667344 +0000 2023-10-30T12:08:25.902 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-10-30 11:57:13.712858447 +0000 2023-10-30T12:08:25.902 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-10-30 11:57:13.712858447 +0000 2023-10-30T12:08:25.903 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2023-10-30 11:57:13.712858447 +0000 2023-10-30T12:08:25.903 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T12:08:25.968 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-10-30T12:08:25.968 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-10-30T12:08:25.969 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000196037 s, 2.6 MB/s 2023-10-30T12:08:25.970 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T12:08:26.026 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 2023-10-30T12:08:26.083 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T12:08:26.083 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:08:26.083 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-10-30T12:08:26.084 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:08:26.084 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:08:26.084 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-10-30 11:59:58.279667344 +0000 2023-10-30T12:08:26.084 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-10-30 11:57:14.205844028 +0000 2023-10-30T12:08:26.084 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-10-30 11:57:14.205844028 +0000 2023-10-30T12:08:26.084 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2023-10-30 11:57:14.205844028 +0000 2023-10-30T12:08:26.085 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T12:08:26.150 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-10-30T12:08:26.151 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-10-30T12:08:26.151 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000227429 s, 2.3 MB/s 2023-10-30T12:08:26.152 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T12:08:26.208 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 2023-10-30T12:08:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T12:08:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:08:26.264 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-10-30T12:08:26.264 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:08:26.265 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:08:26.265 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-10-30 11:59:58.279667344 +0000 2023-10-30T12:08:26.265 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-10-30 11:57:14.703829462 +0000 2023-10-30T12:08:26.265 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-10-30 11:57:14.703829462 +0000 2023-10-30T12:08:26.265 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2023-10-30 11:57:14.703829462 +0000 2023-10-30T12:08:26.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T12:08:26.331 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-10-30T12:08:26.331 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-10-30T12:08:26.331 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000275258 s, 1.9 MB/s 2023-10-30T12:08:26.332 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T12:08:26.389 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 2023-10-30T12:08:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T12:08:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:08:26.444 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-10-30T12:08:26.444 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:08:26.445 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:08:26.445 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-10-30 11:59:58.279667344 +0000 2023-10-30T12:08:26.445 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-10-30 11:57:15.200814925 +0000 2023-10-30T12:08:26.445 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-10-30 11:57:15.200814925 +0000 2023-10-30T12:08:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2023-10-30 11:57:15.200814925 +0000 2023-10-30T12:08:26.446 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T12:08:26.509 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-10-30T12:08:26.510 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-10-30T12:08:26.510 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000243485 s, 2.1 MB/s 2023-10-30T12:08:26.511 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T12:08:26.567 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:08:26.567 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T12:08:26.567 INFO:tasks.ceph:Generating config... 2023-10-30T12:08:26.569 INFO:tasks.ceph:[client] debug monc = 20 2023-10-30T12:08:26.569 INFO:tasks.ceph:[client] debug ms = 1 2023-10-30T12:08:26.569 INFO:tasks.ceph:[client] debug objecter = 20 2023-10-30T12:08:26.570 INFO:tasks.ceph:[client] debug rados = 20 2023-10-30T12:08:26.570 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-30T12:08:26.570 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-10-30T12:08:26.570 INFO:tasks.ceph:[global] ms bind msgr1 = False 2023-10-30T12:08:26.571 INFO:tasks.ceph:[global] ms bind msgr2 = True 2023-10-30T12:08:26.571 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-30T12:08:26.571 INFO:tasks.ceph:[global] ms type = async 2023-10-30T12:08:26.571 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-10-30T12:08:26.572 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-30T12:08:26.572 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-30T12:08:26.572 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-30T12:08:26.572 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-30T12:08:26.573 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-30T12:08:26.573 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-10-30T12:08:26.573 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-10-30T12:08:26.573 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-10-30T12:08:26.574 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-10-30T12:08:26.574 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-10-30T12:08:26.574 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-10-30T12:08:26.575 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-10-30T12:08:26.575 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-10-30T12:08:26.575 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-30T12:08:26.575 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2023-10-30T12:08:26.576 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-30T12:08:26.576 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-30T12:08:26.576 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-10-30T12:08:26.576 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-10-30T12:08:26.577 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-10-30T12:08:26.577 INFO:tasks.ceph:[osd] debug monc = 20 2023-10-30T12:08:26.577 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-30T12:08:26.577 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-30T12:08:26.578 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-30T12:08:26.578 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-30T12:08:26.578 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-30T12:08:26.579 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-30T12:08:26.579 INFO:tasks.ceph:[osd] osd class default list = * 2023-10-30T12:08:26.579 INFO:tasks.ceph:[osd] osd class load list = * 2023-10-30T12:08:26.579 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2023-10-30T12:08:26.580 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-10-30T12:08:26.580 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-10-30T12:08:26.580 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-30T12:08:26.580 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-10-30T12:08:26.581 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-10-30T12:08:26.581 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-10-30T12:08:26.581 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-30T12:08:26.581 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-10-30T12:08:26.582 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-10-30T12:08:26.582 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-30T12:08:26.582 INFO:tasks.ceph:Setting up mon.a... 2023-10-30T12:08:26.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-30T12:08:26.643 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.keyring 2023-10-30T12:08:26.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-30T12:08:26.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T12:08:26.729 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.3'), ('mon.c', '[v2:172.21.15.3:3301,v1:172.21.15.3:6790]'), ('mon.b', '172.21.15.92')] 2023-10-30T12:08:26.730 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.3,[v2:172.21.15.3:3301,v1:172.21.15.3:6790],172.21.15.92', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-10-30T12:08:26.731 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:26.731 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T12:08:26.784 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.3 --addv c '[v2:172.21.15.3:3301,v1:172.21.15.3:6790]' --add b 172.21.15.92 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:08:26.870 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:08:26.870 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: generated fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:08:26.871 INFO:teuthology.orchestra.run.smithi003.stdout:setting min_mon_release = pacific 2023-10-30T12:08:26.871 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 0 2023-10-30T12:08:26.871 INFO:teuthology.orchestra.run.smithi003.stdout:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:08:26.871 INFO:teuthology.orchestra.run.smithi003.stdout:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:08:26.871 INFO:teuthology.orchestra.run.smithi003.stdout:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:08:26.871 INFO:teuthology.orchestra.run.smithi003.stdout:min_mon_release 16 (pacific) 2023-10-30T12:08:26.872 INFO:teuthology.orchestra.run.smithi003.stdout:election_strategy: 3 2023-10-30T12:08:26.872 INFO:teuthology.orchestra.run.smithi003.stdout:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:08:26.872 INFO:teuthology.orchestra.run.smithi003.stdout:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:08:26.872 INFO:teuthology.orchestra.run.smithi003.stdout:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:08:26.872 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-30T12:08:26.874 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T12:08:26.889 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9bae44ae-7379-4ab2-a454-4d550607110c... 2023-10-30T12:08:26.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T12:08:26.931 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T12:08:26.982 INFO:teuthology.orchestra.run.smithi092.stdout:[global] 2023-10-30T12:08:26.983 INFO:teuthology.orchestra.run.smithi092.stdout: chdir = "" 2023-10-30T12:08:26.983 INFO:teuthology.orchestra.run.smithi092.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T12:08:26.983 INFO:teuthology.orchestra.run.smithi092.stdout: auth supported = cephx 2023-10-30T12:08:26.983 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.984 INFO:teuthology.orchestra.run.smithi092.stdout: filestore xattr use omap = true 2023-10-30T12:08:26.984 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.984 INFO:teuthology.orchestra.run.smithi092.stdout: mon clock drift allowed = 1.000 2023-10-30T12:08:26.984 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.984 INFO:teuthology.orchestra.run.smithi092.stdout: osd crush chooseleaf type = 0 2023-10-30T12:08:26.985 INFO:teuthology.orchestra.run.smithi092.stdout: auth debug = true 2023-10-30T12:08:26.985 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.985 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on old message = true 2023-10-30T12:08:26.985 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on bug = true 2023-10-30T12:08:26.985 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.986 INFO:teuthology.orchestra.run.smithi092.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T12:08:26.986 INFO:teuthology.orchestra.run.smithi092.stdout: mon pg warn max object skew = 0 2023-10-30T12:08:26.986 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.986 INFO:teuthology.orchestra.run.smithi092.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T12:08:26.986 INFO:teuthology.orchestra.run.smithi092.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T12:08:26.987 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.987 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default size = 2 2023-10-30T12:08:26.987 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.987 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow primary affinity = true 2023-10-30T12:08:26.987 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow pg remap = true 2023-10-30T12:08:26.988 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on legacy crush tunables = false 2023-10-30T12:08:26.988 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on crush straw calc version zero = false 2023-10-30T12:08:26.988 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on no sortbitwise = false 2023-10-30T12:08:26.988 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on osd down out interval zero = false 2023-10-30T12:08:26.988 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on too few osds = false 2023-10-30T12:08:26.989 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T12:08:26.989 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T12:08:26.989 INFO:teuthology.orchestra.run.smithi092.stdout: mon_allow_pool_size_one = true 2023-10-30T12:08:26.989 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.989 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T12:08:26.990 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.990 INFO:teuthology.orchestra.run.smithi092.stdout: osd default data pool replay window = 5 2023-10-30T12:08:26.990 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.991 INFO:teuthology.orchestra.run.smithi092.stdout: mon allow pool delete = true 2023-10-30T12:08:26.991 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.991 INFO:teuthology.orchestra.run.smithi092.stdout: mon cluster log file level = debug 2023-10-30T12:08:26.991 INFO:teuthology.orchestra.run.smithi092.stdout: debug asserts on shutdown = true 2023-10-30T12:08:26.991 INFO:teuthology.orchestra.run.smithi092.stdout: mon health detail to clog = false 2023-10-30T12:08:26.992 INFO:teuthology.orchestra.run.smithi092.stdout: mon host = "172.21.15.3,[v2:172.21.15.3:3301,v1:172.21.15.3:6790],172.21.15.92" 2023-10-30T12:08:26.992 INFO:teuthology.orchestra.run.smithi092.stdout: mon client directed command retry = 5 2023-10-30T12:08:26.992 INFO:teuthology.orchestra.run.smithi092.stdout: mon election default strategy = 3 2023-10-30T12:08:26.992 INFO:teuthology.orchestra.run.smithi092.stdout: ms bind msgr1 = False 2023-10-30T12:08:26.992 INFO:teuthology.orchestra.run.smithi092.stdout: ms bind msgr2 = True 2023-10-30T12:08:26.993 INFO:teuthology.orchestra.run.smithi092.stdout: ms inject socket failures = 5000 2023-10-30T12:08:26.993 INFO:teuthology.orchestra.run.smithi092.stdout: ms type = async 2023-10-30T12:08:26.993 INFO:teuthology.orchestra.run.smithi092.stdout: osd heartbeat grace = 80 2023-10-30T12:08:26.993 INFO:teuthology.orchestra.run.smithi092.stdout: fsid = 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:08:26.993 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.993 INFO:teuthology.orchestra.run.smithi092.stdout:[osd] 2023-10-30T12:08:26.994 INFO:teuthology.orchestra.run.smithi092.stdout: osd journal size = 100 2023-10-30T12:08:26.994 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.994 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub load threshold = 5.0 2023-10-30T12:08:26.994 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub max interval = 600 2023-10-30T12:08:26.994 INFO:teuthology.orchestra.run.smithi092.stdout: osd mclock profile = high_recovery_ops 2023-10-30T12:08:26.995 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.995 INFO:teuthology.orchestra.run.smithi092.stdout: osd recover clone overlap = true 2023-10-30T12:08:26.995 INFO:teuthology.orchestra.run.smithi092.stdout: osd recovery max chunk = 1048576 2023-10-30T12:08:26.995 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.995 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug shutdown = true 2023-10-30T12:08:26.996 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug op order = true 2023-10-30T12:08:26.996 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify stray on activate = true 2023-10-30T12:08:26.996 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.996 INFO:teuthology.orchestra.run.smithi092.stdout: osd open classes on start = true 2023-10-30T12:08:26.996 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug pg log writeout = true 2023-10-30T12:08:26.997 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.997 INFO:teuthology.orchestra.run.smithi092.stdout: osd deep scrub update digest min age = 30 2023-10-30T12:08:26.997 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.997 INFO:teuthology.orchestra.run.smithi092.stdout: osd map max advance = 10 2023-10-30T12:08:26.997 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.997 INFO:teuthology.orchestra.run.smithi092.stdout: journal zero on create = true 2023-10-30T12:08:26.998 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.998 INFO:teuthology.orchestra.run.smithi092.stdout: filestore ondisk finisher threads = 3 2023-10-30T12:08:26.998 INFO:teuthology.orchestra.run.smithi092.stdout: filestore apply finisher threads = 3 2023-10-30T12:08:26.998 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:26.998 INFO:teuthology.orchestra.run.smithi092.stdout: bdev debug aio = true 2023-10-30T12:08:26.999 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug misdirected ops = true 2023-10-30T12:08:26.999 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore block size = 96636764160 2023-10-30T12:08:26.999 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore compression algorithm = lz4 2023-10-30T12:08:26.999 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore compression mode = aggressive 2023-10-30T12:08:26.999 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore fsck on mount = True 2023-10-30T12:08:27.000 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore zero block detection = True 2023-10-30T12:08:27.000 INFO:teuthology.orchestra.run.smithi092.stdout: debug bluefs = 1 2023-10-30T12:08:27.000 INFO:teuthology.orchestra.run.smithi092.stdout: debug bluestore = 1 2023-10-30T12:08:27.000 INFO:teuthology.orchestra.run.smithi092.stdout: debug monc = 20 2023-10-30T12:08:27.000 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2023-10-30T12:08:27.001 INFO:teuthology.orchestra.run.smithi092.stdout: debug osd = 20 2023-10-30T12:08:27.001 INFO:teuthology.orchestra.run.smithi092.stdout: debug rocksdb = 10 2023-10-30T12:08:27.001 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T12:08:27.001 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd full ratio = 0.9 2023-10-30T12:08:27.001 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd nearfull ratio = 0.8 2023-10-30T12:08:27.001 INFO:teuthology.orchestra.run.smithi092.stdout: osd class default list = * 2023-10-30T12:08:27.002 INFO:teuthology.orchestra.run.smithi092.stdout: osd class load list = * 2023-10-30T12:08:27.002 INFO:teuthology.orchestra.run.smithi092.stdout: osd client watch timeout = 120 2023-10-30T12:08:27.002 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify cached snaps = True 2023-10-30T12:08:27.002 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify missing on start = True 2023-10-30T12:08:27.002 INFO:teuthology.orchestra.run.smithi092.stdout: osd failsafe full ratio = 0.95 2023-10-30T12:08:27.003 INFO:teuthology.orchestra.run.smithi092.stdout: osd fast shutdown = False 2023-10-30T12:08:27.003 INFO:teuthology.orchestra.run.smithi092.stdout: osd mclock override recovery settings = True 2023-10-30T12:08:27.003 INFO:teuthology.orchestra.run.smithi092.stdout: osd objectstore = bluestore 2023-10-30T12:08:27.003 INFO:teuthology.orchestra.run.smithi092.stdout: osd op queue = debug_random 2023-10-30T12:08:27.003 INFO:teuthology.orchestra.run.smithi092.stdout: osd op queue cut off = debug_random 2023-10-30T12:08:27.004 INFO:teuthology.orchestra.run.smithi092.stdout: osd shutdown pgref assert = True 2023-10-30T12:08:27.004 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:27.004 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr] 2023-10-30T12:08:27.004 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2023-10-30T12:08:27.004 INFO:teuthology.orchestra.run.smithi092.stdout: debug mgr = 20 2023-10-30T12:08:27.005 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2023-10-30T12:08:27.005 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2023-10-30T12:08:27.005 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:08:27.005 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:08:27.005 INFO:teuthology.orchestra.run.smithi092.stdout: mgr/telemetry/nag = false 2023-10-30T12:08:27.006 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:27.006 INFO:teuthology.orchestra.run.smithi092.stdout:[mon] 2023-10-30T12:08:27.006 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2023-10-30T12:08:27.006 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2023-10-30T12:08:27.006 INFO:teuthology.orchestra.run.smithi092.stdout: debug paxos = 20 2023-10-30T12:08:27.007 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2023-10-30T12:08:27.007 INFO:teuthology.orchestra.run.smithi092.stdout: mon data avail warn = 5 2023-10-30T12:08:27.007 INFO:teuthology.orchestra.run.smithi092.stdout: mon mgr mkfs grace = 240 2023-10-30T12:08:27.007 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:08:27.007 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd reporter subtree level = osd 2023-10-30T12:08:27.007 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd prime pg temp = true 2023-10-30T12:08:27.008 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:08:27.008 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:27.008 INFO:teuthology.orchestra.run.smithi092.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T12:08:27.008 INFO:teuthology.orchestra.run.smithi092.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T12:08:27.008 INFO:teuthology.orchestra.run.smithi092.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T12:08:27.009 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:27.009 INFO:teuthology.orchestra.run.smithi092.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T12:08:27.009 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T12:08:27.009 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T12:08:27.009 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:27.009 INFO:teuthology.orchestra.run.smithi092.stdout: # 1m isn't quite enough 2023-10-30T12:08:27.010 INFO:teuthology.orchestra.run.smithi092.stdout: mon_down_mkfs_grace = 2m 2023-10-30T12:08:27.010 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:27.010 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_filestore_osds = false 2023-10-30T12:08:27.010 INFO:teuthology.orchestra.run.smithi092.stdout: mon min osdmap epochs = 50 2023-10-30T12:08:27.010 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd crush smoke test = False 2023-10-30T12:08:27.011 INFO:teuthology.orchestra.run.smithi092.stdout: mon osdmap full prune interval = 2 2023-10-30T12:08:27.011 INFO:teuthology.orchestra.run.smithi092.stdout: mon osdmap full prune min = 15 2023-10-30T12:08:27.011 INFO:teuthology.orchestra.run.smithi092.stdout: mon osdmap full prune txsize = 2 2023-10-30T12:08:27.011 INFO:teuthology.orchestra.run.smithi092.stdout: mon scrub interval = 300 2023-10-30T12:08:27.011 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on pool no app = False 2023-10-30T12:08:27.012 INFO:teuthology.orchestra.run.smithi092.stdout: paxos service trim min = 10 2023-10-30T12:08:27.012 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-10-30T12:08:27.012 INFO:teuthology.orchestra.run.smithi092.stdout:[client] 2023-10-30T12:08:27.012 INFO:teuthology.orchestra.run.smithi092.stdout: rgw cache enabled = true 2023-10-30T12:08:27.012 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable ops log = true 2023-10-30T12:08:27.013 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable usage log = true 2023-10-30T12:08:27.013 INFO:teuthology.orchestra.run.smithi092.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T12:08:27.013 INFO:teuthology.orchestra.run.smithi092.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T12:08:27.013 INFO:teuthology.orchestra.run.smithi092.stdout: debug monc = 20 2023-10-30T12:08:27.013 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2023-10-30T12:08:27.014 INFO:teuthology.orchestra.run.smithi092.stdout: debug objecter = 20 2023-10-30T12:08:27.014 INFO:teuthology.orchestra.run.smithi092.stdout: debug rados = 20 2023-10-30T12:08:27.014 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.a] 2023-10-30T12:08:27.014 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.c] 2023-10-30T12:08:27.014 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.b] 2023-10-30T12:08:27.016 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2023-10-30T12:08:27.016 INFO:teuthology.orchestra.run.smithi003.stdout: chdir = "" 2023-10-30T12:08:27.016 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T12:08:27.017 INFO:teuthology.orchestra.run.smithi003.stdout: auth supported = cephx 2023-10-30T12:08:27.017 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.017 INFO:teuthology.orchestra.run.smithi003.stdout: filestore xattr use omap = true 2023-10-30T12:08:27.017 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.017 INFO:teuthology.orchestra.run.smithi003.stdout: mon clock drift allowed = 1.000 2023-10-30T12:08:27.018 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.018 INFO:teuthology.orchestra.run.smithi003.stdout: osd crush chooseleaf type = 0 2023-10-30T12:08:27.018 INFO:teuthology.orchestra.run.smithi003.stdout: auth debug = true 2023-10-30T12:08:27.018 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.018 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on old message = true 2023-10-30T12:08:27.019 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on bug = true 2023-10-30T12:08:27.019 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.019 INFO:teuthology.orchestra.run.smithi003.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T12:08:27.019 INFO:teuthology.orchestra.run.smithi003.stdout: mon pg warn max object skew = 0 2023-10-30T12:08:27.019 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.020 INFO:teuthology.orchestra.run.smithi003.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T12:08:27.020 INFO:teuthology.orchestra.run.smithi003.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T12:08:27.020 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.020 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default size = 2 2023-10-30T12:08:27.020 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.021 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow primary affinity = true 2023-10-30T12:08:27.021 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow pg remap = true 2023-10-30T12:08:27.021 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on legacy crush tunables = false 2023-10-30T12:08:27.021 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on crush straw calc version zero = false 2023-10-30T12:08:27.021 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on no sortbitwise = false 2023-10-30T12:08:27.022 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on osd down out interval zero = false 2023-10-30T12:08:27.022 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on too few osds = false 2023-10-30T12:08:27.022 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T12:08:27.022 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T12:08:27.022 INFO:teuthology.orchestra.run.smithi003.stdout: mon_allow_pool_size_one = true 2023-10-30T12:08:27.023 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.023 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T12:08:27.023 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.025 INFO:teuthology.orchestra.run.smithi003.stdout: osd default data pool replay window = 5 2023-10-30T12:08:27.025 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.025 INFO:teuthology.orchestra.run.smithi003.stdout: mon allow pool delete = true 2023-10-30T12:08:27.026 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.026 INFO:teuthology.orchestra.run.smithi003.stdout: mon cluster log file level = debug 2023-10-30T12:08:27.026 INFO:teuthology.orchestra.run.smithi003.stdout: debug asserts on shutdown = true 2023-10-30T12:08:27.026 INFO:teuthology.orchestra.run.smithi003.stdout: mon health detail to clog = false 2023-10-30T12:08:27.026 INFO:teuthology.orchestra.run.smithi003.stdout: mon host = "172.21.15.3,[v2:172.21.15.3:3301,v1:172.21.15.3:6790],172.21.15.92" 2023-10-30T12:08:27.027 INFO:teuthology.orchestra.run.smithi003.stdout: mon client directed command retry = 5 2023-10-30T12:08:27.027 INFO:teuthology.orchestra.run.smithi003.stdout: mon election default strategy = 3 2023-10-30T12:08:27.027 INFO:teuthology.orchestra.run.smithi003.stdout: ms bind msgr1 = False 2023-10-30T12:08:27.027 INFO:teuthology.orchestra.run.smithi003.stdout: ms bind msgr2 = True 2023-10-30T12:08:27.027 INFO:teuthology.orchestra.run.smithi003.stdout: ms inject socket failures = 5000 2023-10-30T12:08:27.028 INFO:teuthology.orchestra.run.smithi003.stdout: ms type = async 2023-10-30T12:08:27.028 INFO:teuthology.orchestra.run.smithi003.stdout: osd heartbeat grace = 80 2023-10-30T12:08:27.028 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:08:27.028 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.028 INFO:teuthology.orchestra.run.smithi003.stdout:[osd] 2023-10-30T12:08:27.029 INFO:teuthology.orchestra.run.smithi003.stdout: osd journal size = 100 2023-10-30T12:08:27.029 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.029 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub load threshold = 5.0 2023-10-30T12:08:27.029 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub max interval = 600 2023-10-30T12:08:27.029 INFO:teuthology.orchestra.run.smithi003.stdout: osd mclock profile = high_recovery_ops 2023-10-30T12:08:27.030 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.030 INFO:teuthology.orchestra.run.smithi003.stdout: osd recover clone overlap = true 2023-10-30T12:08:27.030 INFO:teuthology.orchestra.run.smithi003.stdout: osd recovery max chunk = 1048576 2023-10-30T12:08:27.030 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.030 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug shutdown = true 2023-10-30T12:08:27.030 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug op order = true 2023-10-30T12:08:27.031 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify stray on activate = true 2023-10-30T12:08:27.031 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.031 INFO:teuthology.orchestra.run.smithi003.stdout: osd open classes on start = true 2023-10-30T12:08:27.031 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug pg log writeout = true 2023-10-30T12:08:27.031 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.032 INFO:teuthology.orchestra.run.smithi003.stdout: osd deep scrub update digest min age = 30 2023-10-30T12:08:27.032 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.032 INFO:teuthology.orchestra.run.smithi003.stdout: osd map max advance = 10 2023-10-30T12:08:27.032 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.032 INFO:teuthology.orchestra.run.smithi003.stdout: journal zero on create = true 2023-10-30T12:08:27.033 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.033 INFO:teuthology.orchestra.run.smithi003.stdout: filestore ondisk finisher threads = 3 2023-10-30T12:08:27.033 INFO:teuthology.orchestra.run.smithi003.stdout: filestore apply finisher threads = 3 2023-10-30T12:08:27.033 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.033 INFO:teuthology.orchestra.run.smithi003.stdout: bdev debug aio = true 2023-10-30T12:08:27.034 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug misdirected ops = true 2023-10-30T12:08:27.034 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore block size = 96636764160 2023-10-30T12:08:27.034 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore compression algorithm = lz4 2023-10-30T12:08:27.034 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore compression mode = aggressive 2023-10-30T12:08:27.034 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore fsck on mount = True 2023-10-30T12:08:27.034 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore zero block detection = True 2023-10-30T12:08:27.035 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluefs = 1 2023-10-30T12:08:27.035 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluestore = 1 2023-10-30T12:08:27.035 INFO:teuthology.orchestra.run.smithi003.stdout: debug monc = 20 2023-10-30T12:08:27.035 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2023-10-30T12:08:27.035 INFO:teuthology.orchestra.run.smithi003.stdout: debug osd = 20 2023-10-30T12:08:27.036 INFO:teuthology.orchestra.run.smithi003.stdout: debug rocksdb = 10 2023-10-30T12:08:27.036 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T12:08:27.036 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd full ratio = 0.9 2023-10-30T12:08:27.036 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd nearfull ratio = 0.8 2023-10-30T12:08:27.036 INFO:teuthology.orchestra.run.smithi003.stdout: osd class default list = * 2023-10-30T12:08:27.037 INFO:teuthology.orchestra.run.smithi003.stdout: osd class load list = * 2023-10-30T12:08:27.037 INFO:teuthology.orchestra.run.smithi003.stdout: osd client watch timeout = 120 2023-10-30T12:08:27.037 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify cached snaps = True 2023-10-30T12:08:27.037 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify missing on start = True 2023-10-30T12:08:27.037 INFO:teuthology.orchestra.run.smithi003.stdout: osd failsafe full ratio = 0.95 2023-10-30T12:08:27.038 INFO:teuthology.orchestra.run.smithi003.stdout: osd fast shutdown = False 2023-10-30T12:08:27.038 INFO:teuthology.orchestra.run.smithi003.stdout: osd mclock override recovery settings = True 2023-10-30T12:08:27.038 INFO:teuthology.orchestra.run.smithi003.stdout: osd objectstore = bluestore 2023-10-30T12:08:27.038 INFO:teuthology.orchestra.run.smithi003.stdout: osd op queue = debug_random 2023-10-30T12:08:27.038 INFO:teuthology.orchestra.run.smithi003.stdout: osd op queue cut off = debug_random 2023-10-30T12:08:27.039 INFO:teuthology.orchestra.run.smithi003.stdout: osd shutdown pgref assert = True 2023-10-30T12:08:27.039 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.039 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr] 2023-10-30T12:08:27.039 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2023-10-30T12:08:27.039 INFO:teuthology.orchestra.run.smithi003.stdout: debug mgr = 20 2023-10-30T12:08:27.040 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2023-10-30T12:08:27.040 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2023-10-30T12:08:27.040 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:08:27.040 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:08:27.040 INFO:teuthology.orchestra.run.smithi003.stdout: mgr/telemetry/nag = false 2023-10-30T12:08:27.040 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.041 INFO:teuthology.orchestra.run.smithi003.stdout:[mon] 2023-10-30T12:08:27.041 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2023-10-30T12:08:27.041 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2023-10-30T12:08:27.041 INFO:teuthology.orchestra.run.smithi003.stdout: debug paxos = 20 2023-10-30T12:08:27.041 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2023-10-30T12:08:27.042 INFO:teuthology.orchestra.run.smithi003.stdout: mon data avail warn = 5 2023-10-30T12:08:27.042 INFO:teuthology.orchestra.run.smithi003.stdout: mon mgr mkfs grace = 240 2023-10-30T12:08:27.042 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:08:27.042 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd reporter subtree level = osd 2023-10-30T12:08:27.042 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd prime pg temp = true 2023-10-30T12:08:27.043 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:08:27.043 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.043 INFO:teuthology.orchestra.run.smithi003.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T12:08:27.043 INFO:teuthology.orchestra.run.smithi003.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T12:08:27.043 INFO:teuthology.orchestra.run.smithi003.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T12:08:27.044 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.044 INFO:teuthology.orchestra.run.smithi003.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T12:08:27.044 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T12:08:27.044 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T12:08:27.044 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.045 INFO:teuthology.orchestra.run.smithi003.stdout: # 1m isn't quite enough 2023-10-30T12:08:27.045 INFO:teuthology.orchestra.run.smithi003.stdout: mon_down_mkfs_grace = 2m 2023-10-30T12:08:27.045 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.045 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_filestore_osds = false 2023-10-30T12:08:27.045 INFO:teuthology.orchestra.run.smithi003.stdout: mon min osdmap epochs = 50 2023-10-30T12:08:27.046 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd crush smoke test = False 2023-10-30T12:08:27.046 INFO:teuthology.orchestra.run.smithi003.stdout: mon osdmap full prune interval = 2 2023-10-30T12:08:27.046 INFO:teuthology.orchestra.run.smithi003.stdout: mon osdmap full prune min = 15 2023-10-30T12:08:27.046 INFO:teuthology.orchestra.run.smithi003.stdout: mon osdmap full prune txsize = 2 2023-10-30T12:08:27.046 INFO:teuthology.orchestra.run.smithi003.stdout: mon scrub interval = 300 2023-10-30T12:08:27.047 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on pool no app = False 2023-10-30T12:08:27.047 INFO:teuthology.orchestra.run.smithi003.stdout: paxos service trim min = 10 2023-10-30T12:08:27.047 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:08:27.047 INFO:teuthology.orchestra.run.smithi003.stdout:[client] 2023-10-30T12:08:27.047 INFO:teuthology.orchestra.run.smithi003.stdout: rgw cache enabled = true 2023-10-30T12:08:27.047 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable ops log = true 2023-10-30T12:08:27.048 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable usage log = true 2023-10-30T12:08:27.048 INFO:teuthology.orchestra.run.smithi003.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T12:08:27.048 INFO:teuthology.orchestra.run.smithi003.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T12:08:27.048 INFO:teuthology.orchestra.run.smithi003.stdout: debug monc = 20 2023-10-30T12:08:27.048 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2023-10-30T12:08:27.049 INFO:teuthology.orchestra.run.smithi003.stdout: debug objecter = 20 2023-10-30T12:08:27.049 INFO:teuthology.orchestra.run.smithi003.stdout: debug rados = 20 2023-10-30T12:08:27.049 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.a] 2023-10-30T12:08:27.049 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.c] 2023-10-30T12:08:27.049 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.b] 2023-10-30T12:08:27.050 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-30T12:08:27.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-30T12:08:27.116 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-30T12:08:27.116 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:27.117 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-30T12:08:27.133 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:27.133 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-30T12:08:27.188 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi003.front.sepia.ceph.com 2023-10-30T12:08:27.188 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:27.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T12:08:27.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T12:08:27.275 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:27.275 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:08:27.330 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi092.front.sepia.ceph.com 2023-10-30T12:08:27.330 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:08:27.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T12:08:27.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T12:08:27.372 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:08:27.372 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:08:27.427 INFO:tasks.ceph:Setting up mon nodes... 2023-10-30T12:08:27.427 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-30T12:08:27.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T12:08:27.490 INFO:teuthology.orchestra.run.smithi003.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T12:08:27.492 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T12:08:27.552 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T12:08:27.555 INFO:tasks.ceph:Setting up mds nodes... 2023-10-30T12:08:27.555 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-30T12:08:27.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-30T12:08:27.602 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-30T12:08:27.621 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-30T12:08:27.664 INFO:teuthology.orchestra.run.smithi092.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-30T12:08:27.679 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-30T12:08:27.679 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T12:08:27.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-30T12:08:27.711 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:08:27.711 INFO:tasks.ceph:role: osd.0 2023-10-30T12:08:27.712 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com 2023-10-30T12:08:27.712 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T12:08:27.789 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:08:27.790 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:08:27.790 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:08:27.790 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:08:27.790 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:08:27.791 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:08:27.791 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:08:27.791 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:08:27.791 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:08:27.791 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:08:28.303 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2023-10-30T12:08:28.305 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2023-10-30T12:08:28.305 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-30T12:08:28.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-30T12:08:28.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-30T12:08:28.578 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:08:28.579 INFO:tasks.ceph:role: osd.1 2023-10-30T12:08:28.579 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com 2023-10-30T12:08:28.579 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T12:08:28.658 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:08:28.659 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:08:28.659 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:08:28.659 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:08:28.659 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:08:28.660 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:08:28.660 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:08:28.660 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:08:28.660 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:08:28.660 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:08:29.110 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2023-10-30T12:08:29.112 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2023-10-30T12:08:29.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-30T12:08:29.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-30T12:08:29.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-30T12:08:29.323 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:08:29.323 INFO:tasks.ceph:role: osd.2 2023-10-30T12:08:29.324 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com 2023-10-30T12:08:29.324 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T12:08:29.404 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:08:29.404 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:08:29.405 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:08:29.405 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:08:29.405 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:08:29.405 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:08:29.405 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:08:29.406 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:08:29.406 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:08:29.406 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:08:29.891 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2023-10-30T12:08:29.893 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2023-10-30T12:08:29.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-30T12:08:30.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-30T12:08:30.087 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-30T12:08:30.160 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:08:30.161 INFO:tasks.ceph:role: osd.3 2023-10-30T12:08:30.161 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi003.front.sepia.ceph.com 2023-10-30T12:08:30.161 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T12:08:30.239 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:08:30.240 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:08:30.240 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:08:30.240 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:08:30.240 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:08:30.241 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:08:30.241 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:08:30.241 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:08:30.241 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:08:30.241 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:08:30.737 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2023-10-30T12:08:30.739 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2023-10-30T12:08:30.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-30T12:08:30.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-30T12:08:30.898 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:08:31.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:31.002+0000 7f44dc3a9500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-30T12:08:31.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:31.003+0000 7f44dc3a9500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-30T12:08:31.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:31.003+0000 7f44dc3a9500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T12:08:31.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:31.003+0000 7f44dc3a9500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T12:08:31.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:31.003+0000 7f44dc3a9500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T12:08:31.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:31.003+0000 7f44dc3a9500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-30T12:08:33.856 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-30T12:08:33.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:08:33.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:33.985+0000 7f5d2702b500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-30T12:08:33.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:33.985+0000 7f5d2702b500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-30T12:08:33.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:33.985+0000 7f5d2702b500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T12:08:33.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:33.985+0000 7f5d2702b500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T12:08:33.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:33.986+0000 7f5d2702b500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T12:08:33.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:33.986+0000 7f5d2702b500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-30T12:08:36.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-30T12:08:36.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:08:36.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:36.959+0000 7f6af80a9500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-30T12:08:36.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:36.959+0000 7f6af80a9500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-30T12:08:36.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:36.959+0000 7f6af80a9500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T12:08:36.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:36.959+0000 7f6af80a9500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T12:08:36.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:36.959+0000 7f6af80a9500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T12:08:36.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:36.960+0000 7f6af80a9500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-30T12:08:39.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-30T12:08:39.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:08:39.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:39.943+0000 7f2555488500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-30T12:08:39.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:39.943+0000 7f2555488500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-30T12:08:39.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:39.943+0000 7f2555488500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T12:08:39.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:39.943+0000 7f2555488500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T12:08:39.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:39.943+0000 7f2555488500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T12:08:39.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:08:39.944+0000 7f2555488500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-30T12:08:42.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-30T12:08:42.821 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-30T12:08:42.852 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:08:42.852 INFO:tasks.ceph:role: osd.4 2023-10-30T12:08:42.853 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com 2023-10-30T12:08:42.853 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T12:08:42.929 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:08:42.930 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:08:42.930 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:08:42.930 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:08:42.930 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:08:42.930 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:08:42.931 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:08:42.931 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:08:42.931 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:08:42.931 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:08:43.443 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2023-10-30T12:08:43.445 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2023-10-30T12:08:43.446 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-30T12:08:43.598 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-30T12:08:43.638 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-30T12:08:43.711 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:08:43.711 INFO:tasks.ceph:role: osd.5 2023-10-30T12:08:43.711 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com 2023-10-30T12:08:43.712 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T12:08:43.786 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:08:43.786 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:08:43.786 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:08:43.787 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:08:43.787 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:08:43.787 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:08:43.787 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:08:43.787 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:08:43.788 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:08:43.788 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:08:44.233 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2023-10-30T12:08:44.234 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2023-10-30T12:08:44.235 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-30T12:08:44.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-30T12:08:44.359 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-30T12:08:44.427 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:08:44.427 INFO:tasks.ceph:role: osd.6 2023-10-30T12:08:44.428 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com 2023-10-30T12:08:44.428 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T12:08:44.501 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:08:44.501 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:08:44.501 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:08:44.502 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:08:44.502 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:08:44.502 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:08:44.502 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:08:44.502 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:08:44.503 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:08:44.503 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:08:44.981 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2023-10-30T12:08:44.983 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2023-10-30T12:08:44.984 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-30T12:08:45.147 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-30T12:08:45.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-30T12:08:45.246 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:08:45.247 INFO:tasks.ceph:role: osd.7 2023-10-30T12:08:45.247 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com 2023-10-30T12:08:45.247 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T12:08:45.319 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:08:45.320 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:08:45.320 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:08:45.320 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:08:45.321 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:08:45.321 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:08:45.321 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:08:45.321 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:08:45.321 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:08:45.322 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:08:45.834 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2023-10-30T12:08:45.836 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2023-10-30T12:08:45.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-30T12:08:45.985 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-30T12:08:46.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:08:46.127 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:46.125+0000 7f7166807500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-30T12:08:46.127 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:46.126+0000 7f7166807500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-30T12:08:46.128 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:46.126+0000 7f7166807500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T12:08:46.128 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:46.126+0000 7f7166807500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T12:08:46.128 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:46.126+0000 7f7166807500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T12:08:46.129 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:46.126+0000 7f7166807500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-30T12:08:48.965 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-30T12:08:48.997 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:08:49.091 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:49.089+0000 7efc8aa13500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-30T12:08:49.091 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:49.090+0000 7efc8aa13500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-30T12:08:49.092 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:49.090+0000 7efc8aa13500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T12:08:49.092 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:49.090+0000 7efc8aa13500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T12:08:49.092 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:49.090+0000 7efc8aa13500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T12:08:49.092 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:49.090+0000 7efc8aa13500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-30T12:08:51.937 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-30T12:08:51.968 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:08:52.060 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:52.058+0000 7fcb3160f500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-30T12:08:52.061 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:52.058+0000 7fcb3160f500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-30T12:08:52.061 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:52.058+0000 7fcb3160f500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T12:08:52.061 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:52.058+0000 7fcb3160f500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T12:08:52.062 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:52.058+0000 7fcb3160f500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T12:08:52.062 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:52.059+0000 7fcb3160f500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-30T12:08:54.905 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-30T12:08:54.932 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:08:55.018 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:55.016+0000 7f32d8449500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-30T12:08:55.018 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:55.016+0000 7f32d8449500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-30T12:08:55.018 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:55.016+0000 7f32d8449500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T12:08:55.019 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:55.016+0000 7f32d8449500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T12:08:55.019 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:55.016+0000 7f32d8449500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T12:08:55.019 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:08:55.017+0000 7f32d8449500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-30T12:08:57.856 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-30T12:08:57.887 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-30T12:08:57.888 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:57.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-30T12:08:57.925 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:57.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-30T12:08:57.999 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:58.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-30T12:08:58.073 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:58.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-30T12:08:58.145 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:58.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-30T12:08:58.218 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:08:58.219 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-30T12:08:58.250 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:08:58.250 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-30T12:08:58.319 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:08:58.320 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-30T12:08:58.389 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:08:58.389 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-30T12:08:58.457 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:08:58.457 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-30T12:08:58.524 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:08:58.525 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-30T12:08:58.543 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:08:58.543 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-30T12:08:58.582 INFO:tasks.ceph:Adding keys to all mons... 2023-10-30T12:08:58.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T12:08:58.585 DEBUG:teuthology.orchestra.run.smithi092:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T12:08:58.625 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.y] 2023-10-30T12:08:58.625 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQC7nD9lrnk1HRAAYB2blWxN3IVnAYk6+0+0NA== 2023-10-30T12:08:58.626 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.0] 2023-10-30T12:08:58.626 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQC/nD9lSx07ABAA/0jFOE5yuNDvvxj0HnbUiA== 2023-10-30T12:08:58.626 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.1] 2023-10-30T12:08:58.626 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDBnD9llM/OOhAAXRZTxcIFjR5xqeWqg8GtsQ== 2023-10-30T12:08:58.627 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.2] 2023-10-30T12:08:58.627 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDEnD9llZZAORAAWHZ2eVU/RroOxXuuk89wHA== 2023-10-30T12:08:58.627 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.3] 2023-10-30T12:08:58.627 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDHnD9lHF5KOBAAby9QpEIIQVXMyJTnsqs/xA== 2023-10-30T12:08:58.627 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.x] 2023-10-30T12:08:58.628 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQC7nD9l+7XsIBAAnwBHZMsB9/3g+Y2Sz8cncQ== 2023-10-30T12:08:58.628 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.4] 2023-10-30T12:08:58.628 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDOnD9llfqKBxAAlgLHIkwLFK4R5pTJV3Ad2Q== 2023-10-30T12:08:58.628 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.5] 2023-10-30T12:08:58.628 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDRnD9l9AdlBRAAhIXahCvE4ENBb8g7lAzoFQ== 2023-10-30T12:08:58.628 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.6] 2023-10-30T12:08:58.629 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDUnD9l3oqHAxAAs3sdxUAYQ5ZwuTsp++zo2w== 2023-10-30T12:08:58.629 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.7] 2023-10-30T12:08:58.629 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDXnD9lkR0KARAAIGdBMGHOSAa7ff35IK798g== 2023-10-30T12:08:58.629 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2023-10-30T12:08:58.629 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQC7nD9lplvkIxAAEfbUNKGRnwASpd9wOW3xoA== 2023-10-30T12:08:58.630 INFO:teuthology.orchestra.run.smithi003.stdout:[client.1] 2023-10-30T12:08:58.630 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQC7nD9lrEyPJxAAbPYXDv4liV36a83Hn2ACfg== 2023-10-30T12:08:58.647 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.y] 2023-10-30T12:08:58.647 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQC7nD9lrnk1HRAAYB2blWxN3IVnAYk6+0+0NA== 2023-10-30T12:08:58.647 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.0] 2023-10-30T12:08:58.648 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQC/nD9lSx07ABAA/0jFOE5yuNDvvxj0HnbUiA== 2023-10-30T12:08:58.648 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.1] 2023-10-30T12:08:58.648 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDBnD9llM/OOhAAXRZTxcIFjR5xqeWqg8GtsQ== 2023-10-30T12:08:58.648 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.2] 2023-10-30T12:08:58.648 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDEnD9llZZAORAAWHZ2eVU/RroOxXuuk89wHA== 2023-10-30T12:08:58.649 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.3] 2023-10-30T12:08:58.649 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDHnD9lHF5KOBAAby9QpEIIQVXMyJTnsqs/xA== 2023-10-30T12:08:58.649 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.x] 2023-10-30T12:08:58.649 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQC7nD9l+7XsIBAAnwBHZMsB9/3g+Y2Sz8cncQ== 2023-10-30T12:08:58.649 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.4] 2023-10-30T12:08:58.650 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDOnD9llfqKBxAAlgLHIkwLFK4R5pTJV3Ad2Q== 2023-10-30T12:08:58.650 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.5] 2023-10-30T12:08:58.650 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDRnD9l9AdlBRAAhIXahCvE4ENBb8g7lAzoFQ== 2023-10-30T12:08:58.650 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.6] 2023-10-30T12:08:58.650 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDUnD9l3oqHAxAAs3sdxUAYQ5ZwuTsp++zo2w== 2023-10-30T12:08:58.651 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.7] 2023-10-30T12:08:58.651 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDXnD9lkR0KARAAIGdBMGHOSAa7ff35IK798g== 2023-10-30T12:08:58.651 INFO:teuthology.orchestra.run.smithi092.stdout:[client.0] 2023-10-30T12:08:58.651 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQC7nD9lplvkIxAAEfbUNKGRnwASpd9wOW3xoA== 2023-10-30T12:08:58.651 INFO:teuthology.orchestra.run.smithi092.stdout:[client.1] 2023-10-30T12:08:58.652 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQC7nD9lrEyPJxAAbPYXDv4liV36a83Hn2ACfg== 2023-10-30T12:08:58.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:08:58.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:08:58.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:58.747 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:58.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:58.805 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:58.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:58.862 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:58.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:58.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:58.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:08:58.973 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:08:59.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:59.028 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:59.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:59.085 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:59.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:59.140 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:59.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:59.196 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:08:59.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T12:08:59.252 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T12:08:59.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T12:08:59.312 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T12:08:59.368 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-30T12:08:59.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-30T12:08:59.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T12:08:59.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-30T12:08:59.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-30T12:08:59.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T12:08:59.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-30T12:08:59.734 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-30T12:08:59.761 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T12:08:59.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-30T12:08:59.884 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:08:59.887 DEBUG:teuthology.orchestra.run.smithi092:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:08:59.939 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-30T12:08:59.939 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-10-30T12:08:59.940 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:08:59.940 DEBUG:teuthology.orchestra.run.smithi003:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-10-30T12:08:59.943 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:08:59.943 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-10-30T12:08:59.944 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T12:08:59.944 DEBUG:teuthology.orchestra.run.smithi003:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-10-30T12:08:59.947 INFO:tasks.ceph.mon.c:Started 2023-10-30T12:08:59.947 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-10-30T12:08:59.948 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T12:08:59.948 DEBUG:teuthology.orchestra.run.smithi092:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-10-30T12:08:59.981 INFO:tasks.ceph.mon.b:Started 2023-10-30T12:08:59.981 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-30T12:08:59.982 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-30T12:08:59.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-10-30T12:08:59.985 INFO:tasks.ceph.mgr.y:Started 2023-10-30T12:08:59.985 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-30T12:08:59.985 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-10-30T12:08:59.987 INFO:tasks.ceph.mgr.x:Started 2023-10-30T12:08:59.988 INFO:tasks.ceph:Setting crush tunables to default 2023-10-30T12:08:59.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd crush tunables default 2023-10-30T12:09:00.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.101+0000 7ff8f9965640 10 monclient: get_monmap_and_config 2023-10-30T12:09:00.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.101+0000 7ff8f9965640 10 monclient: build_initial_monmap 2023-10-30T12:09:00.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.102+0000 7ff8f9965640 10 monclient: monmap: 2023-10-30T12:09:00.104 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:09:00.104 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:00.104 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:09:00.102981+0000 2023-10-30T12:09:00.104 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:09:00.102981+0000 2023-10-30T12:09:00.105 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:00.105 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:00.105 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:09:00.106 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:09:00.106 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:09:00.106 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:09:00.106 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:09:00.106 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:09:00.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.102+0000 7ff8f9965640 1 Processor -- start 2023-10-30T12:09:00.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.102+0000 7ff8f9965640 1 -- start start 2023-10-30T12:09:00.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.102+0000 7ff8f9965640 10 monclient: init 2023-10-30T12:09:00.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f9965640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:00.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f9965640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-10-30T12:09:00.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f9965640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 0x7ff8f4144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:00.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f9965640 10 monclient(hunting): picked mon.noname-b con 0x7ff8f40b4440 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:00.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f9965640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8f41453e0 addr v1:172.21.15.3:6789/0 2023-10-30T12:09:00.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f9965640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 0x7ff8f414c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:00.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f27fc640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7ff8f41453e0 legacy=0x7ff8f40b1b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2023-10-30T12:09:00.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f2ffd640 1 -- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 msgr2=0x7ff8f4144ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3301/0 2023-10-30T12:09:00.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f9965640 10 monclient(hunting): picked mon.noname-a con 0x7ff8f40b2230 addr v2:172.21.15.3:3300/0 2023-10-30T12:09:00.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f37fe640 1 -- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 msgr2=0x7ff8f414c3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3300/0 2023-10-30T12:09:00.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f37fe640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 0x7ff8f414c3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:00.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f2ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 0x7ff8f4144ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:00.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f9965640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f40a9da0 con 0x7ff8f41453e0 2023-10-30T12:09:00.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f9965640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:00.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f9965640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f4067a10 con 0x7ff8f40b2230 2023-10-30T12:09:00.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f9965640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:00.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f9965640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f40aa080 con 0x7ff8f40b4440 2023-10-30T12:09:00.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f9965640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:00.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.103+0000 7ff8f9965640 10 monclient(hunting): authenticate will time out at 1331.317570s 2023-10-30T12:09:00.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.304+0000 7ff8f27fc640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7ff8f41453e0 legacy=0x7ff8f40b1b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2023-10-30T12:09:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.304+0000 7ff8f37fe640 1 -- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 msgr2=0x7ff8f414c3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3300/0 2023-10-30T12:09:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.304+0000 7ff8f2ffd640 1 -- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 msgr2=0x7ff8f4144ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3301/0 2023-10-30T12:09:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.304+0000 7ff8f37fe640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 0x7ff8f414c3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:00.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.304+0000 7ff8f2ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 0x7ff8f4144ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:00.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.705+0000 7ff8f37fe640 1 -- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 msgr2=0x7ff8f414c3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3300/0 2023-10-30T12:09:00.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.705+0000 7ff8f2ffd640 1 -- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 msgr2=0x7ff8f4144ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3301/0 2023-10-30T12:09:00.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.705+0000 7ff8f27fc640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7ff8f41453e0 legacy=0x7ff8f40b1b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2023-10-30T12:09:00.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.705+0000 7ff8f37fe640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 0x7ff8f414c3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:00.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:00.705+0000 7ff8f2ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 0x7ff8f4144ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:01.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:01.506+0000 7ff8f37fe640 1 -- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 msgr2=0x7ff8f414c3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3300/0 2023-10-30T12:09:01.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:01.506+0000 7ff8f2ffd640 1 -- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 msgr2=0x7ff8f4144ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3301/0 2023-10-30T12:09:01.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:01.506+0000 7ff8f27fc640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7ff8f41453e0 legacy=0x7ff8f40b1b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2023-10-30T12:09:01.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:01.506+0000 7ff8f37fe640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 0x7ff8f414c3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:09:01.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:01.506+0000 7ff8f2ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 0x7ff8f4144ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:09:03.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 10 monclient(hunting): tick 2023-10-30T12:09:03.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:09:03.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:09:03.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:09:03.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 1 -- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 msgr2=0x7ff8f4144ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:03.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 0x7ff8f4144ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:03.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 1 -- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 msgr2=0x7ff8f414c3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:03.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 0x7ff8f414c3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:03.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7ff8f41453e0 legacy=0x7ff8f40b1b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:03.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 10 monclient: _add_conns ranks=[4,1] 2023-10-30T12:09:03.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8e00036f0 0x7ff8e0005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:03.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7ff8e00036f0 addr v2:172.21.15.92:3300/0 2023-10-30T12:09:03.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8e0006260 addr v1:172.21.15.92:6789/0 2023-10-30T12:09:03.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f40a9da0 con 0x7ff8e0006260 2023-10-30T12:09:03.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:03.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f4067a10 con 0x7ff8e00036f0 2023-10-30T12:09:03.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f0ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:03.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f27fc640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7ff8e0006260 legacy=0x7ff8e0008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2023-10-30T12:09:03.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f2ffd640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7ff8e00036f0 msgr2=0x7ff8e0005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2023-10-30T12:09:03.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.103+0000 7ff8f2ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8e00036f0 0x7ff8e0005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:03.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.304+0000 7ff8f2ffd640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7ff8e00036f0 msgr2=0x7ff8e0005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2023-10-30T12:09:03.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.304+0000 7ff8f27fc640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7ff8e0006260 legacy=0x7ff8e0008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2023-10-30T12:09:03.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.304+0000 7ff8f2ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8e00036f0 0x7ff8e0005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:03.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.705+0000 7ff8f27fc640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7ff8e0006260 legacy=0x7ff8e0008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2023-10-30T12:09:03.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.705+0000 7ff8f2ffd640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7ff8e00036f0 msgr2=0x7ff8e0005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2023-10-30T12:09:03.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:03.705+0000 7ff8f2ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8e00036f0 0x7ff8e0005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:04.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:04.506+0000 7ff8f2ffd640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7ff8e00036f0 msgr2=0x7ff8e0005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2023-10-30T12:09:04.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:04.506+0000 7ff8f27fc640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7ff8e0006260 legacy=0x7ff8e0008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2023-10-30T12:09:04.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:04.506+0000 7ff8f2ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8e00036f0 0x7ff8e0005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:09:06.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 10 monclient(hunting): tick 2023-10-30T12:09:06.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:09:06.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:09:06.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:09:06.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7ff8e00036f0 msgr2=0x7ff8e0005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:06.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8e00036f0 0x7ff8e0005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:06.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7ff8e0006260 legacy=0x7ff8e0008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:06.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-10-30T12:09:06.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f37fe640 1 -- reap_dead start 2023-10-30T12:09:06.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8f41453e0 addr v1:172.21.15.3:6789/0 2023-10-30T12:09:06.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 0x7ff8e000a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:06.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7ff8f40b2230 addr v2:172.21.15.3:3300/0 2023-10-30T12:09:06.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 0x7ff8e0012290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:06.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7ff8f40b4440 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:06.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f40a9da0 con 0x7ff8f41453e0 2023-10-30T12:09:06.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:06.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f4067a10 con 0x7ff8f40b2230 2023-10-30T12:09:06.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:06.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f40aa080 con 0x7ff8f40b4440 2023-10-30T12:09:06.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f0ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:06.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f2ffd640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7ff8f41453e0 legacy=0x7ff8e0009f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2023-10-30T12:09:06.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f27fc640 1 -- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 msgr2=0x7ff8e000a6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3300/0 2023-10-30T12:09:06.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f27fc640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 0x7ff8e000a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:06.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f37fe640 1 -- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 msgr2=0x7ff8e0012290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3301/0 2023-10-30T12:09:06.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.103+0000 7ff8f37fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 0x7ff8e0012290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:06.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.304+0000 7ff8f2ffd640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7ff8f41453e0 legacy=0x7ff8e0009f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2023-10-30T12:09:06.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.304+0000 7ff8f27fc640 1 -- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 msgr2=0x7ff8e000a6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3300/0 2023-10-30T12:09:06.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.304+0000 7ff8f37fe640 1 -- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 msgr2=0x7ff8e0012290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3301/0 2023-10-30T12:09:06.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.304+0000 7ff8f27fc640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 0x7ff8e000a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:06.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.304+0000 7ff8f37fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 0x7ff8e0012290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:06.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.705+0000 7ff8f2ffd640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7ff8f41453e0 legacy=0x7ff8e0009f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2023-10-30T12:09:06.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.705+0000 7ff8f27fc640 1 -- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 msgr2=0x7ff8e000a6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3300/0 2023-10-30T12:09:06.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.705+0000 7ff8f37fe640 1 -- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 msgr2=0x7ff8e0012290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3301/0 2023-10-30T12:09:06.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.705+0000 7ff8f27fc640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 0x7ff8e000a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:06.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:06.705+0000 7ff8f37fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 0x7ff8e0012290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:07.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:07.506+0000 7ff8f2ffd640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7ff8f41453e0 legacy=0x7ff8e0009f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2023-10-30T12:09:07.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:07.506+0000 7ff8f27fc640 1 -- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 msgr2=0x7ff8e000a6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3300/0 2023-10-30T12:09:07.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:07.506+0000 7ff8f37fe640 1 -- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 msgr2=0x7ff8e0012290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.3:3301/0 2023-10-30T12:09:07.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:07.506+0000 7ff8f27fc640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 0x7ff8e000a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:09:07.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:07.506+0000 7ff8f37fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 0x7ff8e0012290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:09:07.644 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:00:07.552 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:09:07.956 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:00:07.864 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:09:08.190 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:00:08.126 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:09:08.195 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:00:08.131 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:09:08.522 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:00:08.457 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:09:08.527 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:00:08.463 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:09:09.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 10 monclient(hunting): tick 2023-10-30T12:09:09.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:09:09.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:09:09.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:09:09.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 1 -- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 msgr2=0x7ff8e0012290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:09.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8f40b4440 0x7ff8e0012290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:09.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 1 -- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 msgr2=0x7ff8e000a6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:09.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff8f40b2230 0x7ff8e000a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:09.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7ff8f41453e0 legacy=0x7ff8e0009f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:09.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 10 monclient: _add_conns ranks=[4,1] 2023-10-30T12:09:09.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 0x7ff8e0005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:09.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7ff8e0008ef0 addr v2:172.21.15.92:3300/0 2023-10-30T12:09:09.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8e00092d0 addr v1:172.21.15.92:6789/0 2023-10-30T12:09:09.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f40a9da0 con 0x7ff8e00092d0 2023-10-30T12:09:09.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:09.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f4067a10 con 0x7ff8e0008ef0 2023-10-30T12:09:09.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f0ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:09.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f27fc640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7ff8e00092d0 legacy=0x7ff8e0008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2023-10-30T12:09:09.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f2ffd640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 msgr2=0x7ff8e0005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2023-10-30T12:09:09.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.103+0000 7ff8f2ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 0x7ff8e0005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:09.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.444+0000 7ff8f2ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 0x7ff8e0005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:09.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.550+0000 7ff8f2ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 0x7ff8e0005b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48124/0 (socket says 172.21.15.3:48124) 2023-10-30T12:09:09.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.550+0000 7ff8f2ffd640 1 -- 172.21.15.3:0/2168404818 learned_addr learned my addr 172.21.15.3:0/2168404818 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:09:09.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.550+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request con 0x7ff8e0008ef0 auth_method 0 2023-10-30T12:09:09.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.550+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:09.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.550+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:09.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.550+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:09.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.578+0000 7ff8f27fc640 1 -- 172.21.15.3:0/2168404818 >> v1:172.21.15.92:6789/0 conn(0x7ff8e00092d0 legacy=0x7ff8e0008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:09:09.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.578+0000 7ff8f27fc640 1 -- 172.21.15.3:0/2168404818 >> v1:172.21.15.92:6789/0 conn(0x7ff8e00092d0 legacy=0x7ff8e0008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:09.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.578+0000 7ff8f27fc640 1 --1- 172.21.15.3:0/2168404818 >> v1:172.21.15.92:6789/0 conn(0x7ff8e00092d0 0x7ff8e0008720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2023-10-30T12:09:09.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.617+0000 7ff8f2ffd640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 msgr2=0x7ff8e0005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:09:09.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.617+0000 7ff8f2ffd640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 msgr2=0x7ff8e0005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:09.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.617+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 0x7ff8e0005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:09.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:09.617+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 0x7ff8e0005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:10.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.018+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 0x7ff8e0005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:10.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.019+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request con 0x7ff8e0008ef0 auth_method 0 2023-10-30T12:09:10.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.019+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:10.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.019+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:10.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.019+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:10.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.019+0000 7ff8f2ffd640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 msgr2=0x7ff8e0005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:09:10.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.019+0000 7ff8f2ffd640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 msgr2=0x7ff8e0005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:10.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.019+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 0x7ff8e0005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:10.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.019+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 0x7ff8e0005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:10.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.821+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 0x7ff8e0005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:10.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.822+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request con 0x7ff8e0008ef0 auth_method 0 2023-10-30T12:09:10.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.822+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:10.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.822+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:10.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.822+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:10.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.822+0000 7ff8f2ffd640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 msgr2=0x7ff8e0005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:09:10.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.822+0000 7ff8f2ffd640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 msgr2=0x7ff8e0005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:10.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.822+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 0x7ff8e0005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:10.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:10.822+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 0x7ff8e0005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:09:12.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.103+0000 7ff8f0ff9640 10 monclient(hunting): tick 2023-10-30T12:09:12.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.103+0000 7ff8f0ff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:09:12.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.103+0000 7ff8f0ff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:09:12.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.103+0000 7ff8f0ff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:09:12.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.103+0000 7ff8f0ff9640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 msgr2=0x7ff8e0005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:12.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.104+0000 7ff8f0ff9640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0008ef0 0x7ff8e0005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:12.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.104+0000 7ff8f0ff9640 1 -- 172.21.15.3:0/2168404818 >> v1:172.21.15.92:6789/0 conn(0x7ff8e00092d0 legacy=0x7ff8e0008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:12.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.104+0000 7ff8f0ff9640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-10-30T12:09:12.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.104+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 reap_dead start 2023-10-30T12:09:12.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.104+0000 7ff8f0ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8e0009760 addr v1:172.21.15.92:6789/0 2023-10-30T12:09:12.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.104+0000 7ff8f0ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8e000a6c0 addr v1:172.21.15.3:6789/0 2023-10-30T12:09:12.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.104+0000 7ff8f0ff9640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 0x7ff8e0007720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:12.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.104+0000 7ff8f0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7ff8e0005b70 addr v2:172.21.15.3:3300/0 2023-10-30T12:09:12.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.104+0000 7ff8f0ff9640 1 -- 172.21.15.3:0/2168404818 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e0009c10 con 0x7ff8e000a6c0 2023-10-30T12:09:12.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.104+0000 7ff8f0ff9640 1 -- 172.21.15.3:0/2168404818 --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e0008c70 con 0x7ff8e0009760 2023-10-30T12:09:12.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.104+0000 7ff8f0ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:12.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.104+0000 7ff8f0ff9640 1 -- 172.21.15.3:0/2168404818 --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f4067a10 con 0x7ff8e0005b70 2023-10-30T12:09:12.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.104+0000 7ff8f0ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:12.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.115+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 0x7ff8e0007720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:12.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.116+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request con 0x7ff8e0005b70 auth_method 0 2023-10-30T12:09:12.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.116+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:12.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.116+0000 7ff8f37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:12.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.116+0000 7ff8f37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:12.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.123+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 msgr2=0x7ff8e0007720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:09:12.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.123+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 msgr2=0x7ff8e0007720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:12.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.123+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 0x7ff8e0007720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:12.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.124+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 0x7ff8e0007720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:12.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.325+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 0x7ff8e0007720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:12.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.325+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request con 0x7ff8e0005b70 auth_method 0 2023-10-30T12:09:12.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.325+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:12.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.325+0000 7ff8f37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:12.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.325+0000 7ff8f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:12.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.326+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 msgr2=0x7ff8e0007720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:09:12.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.326+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 msgr2=0x7ff8e0007720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:12.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.326+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 0x7ff8e0007720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:12.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.326+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 0x7ff8e0007720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:12.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.727+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 0x7ff8e0007720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:12.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.728+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request con 0x7ff8e0005b70 auth_method 0 2023-10-30T12:09:12.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.728+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:12.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.728+0000 7ff8f37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:12.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.728+0000 7ff8f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:12.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.729+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 msgr2=0x7ff8e0007720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:09:12.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.729+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 msgr2=0x7ff8e0007720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:12.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.729+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 0x7ff8e0007720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:12.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:12.729+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 0x7ff8e0007720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:13.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:13.530+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 0x7ff8e0007720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:13.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:13.531+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request con 0x7ff8e0005b70 auth_method 0 2023-10-30T12:09:13.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:13.531+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:13.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:13.531+0000 7ff8f37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:13.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:13.531+0000 7ff8f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:13.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:13.531+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 msgr2=0x7ff8e0007720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:09:13.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:13.531+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 msgr2=0x7ff8e0007720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:13.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:13.531+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 0x7ff8e0007720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:13.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:13.532+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 0x7ff8e0007720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:09:15.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 10 monclient(hunting): tick 2023-10-30T12:09:15.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:09:15.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:09:15.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:09:15.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 msgr2=0x7ff8e0007720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:15.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.3:3300/0 conn(0x7ff8e0005b70 0x7ff8e0007720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:15.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 1 -- 172.21.15.3:0/2168404818 >> v1:172.21.15.92:6789/0 conn(0x7ff8e0009760 legacy=0x7ff8e00157f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:15.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 1 -- 172.21.15.3:0/2168404818 >> v1:172.21.15.3:6789/0 conn(0x7ff8e000a6c0 legacy=0x7ff8e0009f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:15.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 10 monclient: _add_conns ranks=[4,2] 2023-10-30T12:09:15.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:15.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7ff8e0007c60 addr v2:172.21.15.92:3300/0 2023-10-30T12:09:15.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 0x7ff8e0011280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:15.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7ff8e0008730 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:15.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:15.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 1 -- 172.21.15.3:0/2168404818 --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f4067a10 con 0x7ff8e0007c60 2023-10-30T12:09:15.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:15.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 1 -- 172.21.15.3:0/2168404818 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f40aa080 con 0x7ff8e0008730 2023-10-30T12:09:15.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f0ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:15.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.104+0000 7ff8f27fc640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 0x7ff8e0011280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:15.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.105+0000 7ff8f27fc640 10 monclient(hunting): get_auth_request con 0x7ff8e0008730 auth_method 0 2023-10-30T12:09:15.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.105+0000 7ff8f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:15.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.105+0000 7ff8f27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:15.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.105+0000 7ff8f27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:15.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.106+0000 7ff8f27fc640 1 -- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 msgr2=0x7ff8e0011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:09:15.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.106+0000 7ff8f27fc640 1 -- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 msgr2=0x7ff8e0011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:15.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.106+0000 7ff8f27fc640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 0x7ff8e0011280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:15.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.106+0000 7ff8f27fc640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 0x7ff8e0011280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:15.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.106+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:15.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.106+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request con 0x7ff8e0007c60 auth_method 0 2023-10-30T12:09:15.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.106+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:15.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.106+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:15.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.106+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:15.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.107+0000 7ff8f2ffd640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:09:15.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.107+0000 7ff8f2ffd640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:15.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.107+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:15.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.107+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:15.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.307+0000 7ff8f27fc640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 0x7ff8e0011280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:15.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.307+0000 7ff8f27fc640 10 monclient(hunting): get_auth_request con 0x7ff8e0008730 auth_method 0 2023-10-30T12:09:15.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.307+0000 7ff8f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:15.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.307+0000 7ff8f27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:15.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.307+0000 7ff8f27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.308+0000 7ff8f27fc640 1 -- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 msgr2=0x7ff8e0011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:09:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.308+0000 7ff8f27fc640 1 -- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 msgr2=0x7ff8e0011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.308+0000 7ff8f27fc640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 0x7ff8e0011280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.308+0000 7ff8f27fc640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 0x7ff8e0011280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.308+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.309+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request con 0x7ff8e0007c60 auth_method 0 2023-10-30T12:09:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.309+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.309+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.309+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.310+0000 7ff8f2ffd640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:09:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.310+0000 7ff8f2ffd640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.310+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.310+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:15.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.709+0000 7ff8f27fc640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 0x7ff8e0011280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:15.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.710+0000 7ff8f27fc640 10 monclient(hunting): get_auth_request con 0x7ff8e0008730 auth_method 0 2023-10-30T12:09:15.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.710+0000 7ff8f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:15.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.710+0000 7ff8f27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:15.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.710+0000 7ff8f27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:15.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.710+0000 7ff8f27fc640 1 -- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 msgr2=0x7ff8e0011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:09:15.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.710+0000 7ff8f27fc640 1 -- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 msgr2=0x7ff8e0011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:15.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.710+0000 7ff8f27fc640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 0x7ff8e0011280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:15.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.710+0000 7ff8f27fc640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 0x7ff8e0011280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:15.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.711+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:15.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.712+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request con 0x7ff8e0007c60 auth_method 0 2023-10-30T12:09:15.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.712+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:15.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.712+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:15.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.712+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:15.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.713+0000 7ff8f2ffd640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:09:15.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.713+0000 7ff8f2ffd640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:15.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.713+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:15.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:15.713+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:16.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.511+0000 7ff8f27fc640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 0x7ff8e0011280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:16.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.511+0000 7ff8f27fc640 10 monclient(hunting): get_auth_request con 0x7ff8e0008730 auth_method 0 2023-10-30T12:09:16.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.511+0000 7ff8f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:16.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.511+0000 7ff8f27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:16.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.511+0000 7ff8f27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:16.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.512+0000 7ff8f27fc640 1 -- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 msgr2=0x7ff8e0011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:09:16.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.512+0000 7ff8f27fc640 1 -- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 msgr2=0x7ff8e0011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:16.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.512+0000 7ff8f27fc640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 0x7ff8e0011280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:16.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.512+0000 7ff8f27fc640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 0x7ff8e0011280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:09:16.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.514+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:16.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.514+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request con 0x7ff8e0007c60 auth_method 0 2023-10-30T12:09:16.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.514+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:16.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.514+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:16.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.514+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.515+0000 7ff8f2ffd640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:09:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.515+0000 7ff8f2ffd640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:16.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.515+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:16.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:16.515+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:09:18.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 10 monclient(hunting): tick 2023-10-30T12:09:18.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:09:18.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:09:18.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:09:18.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 1 -- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 msgr2=0x7ff8e0011280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:18.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8e0008730 0x7ff8e0011280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:18.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:18.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:18.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-10-30T12:09:18.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 reap_dead start 2023-10-30T12:09:18.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8e000a6c0 addr v1:172.21.15.3:6789/0 2023-10-30T12:09:18.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8e0009760 addr v1:172.21.15.92:6789/0 2023-10-30T12:09:18.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:18.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7ff8e0007c60 addr v2:172.21.15.92:3300/0 2023-10-30T12:09:18.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 1 -- 172.21.15.3:0/2168404818 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e0015270 con 0x7ff8e000a6c0 2023-10-30T12:09:18.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 1 -- 172.21.15.3:0/2168404818 --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e0005b70 con 0x7ff8e0009760 2023-10-30T12:09:18.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:18.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 1 -- 172.21.15.3:0/2168404818 --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f4067a10 con 0x7ff8e0007c60 2023-10-30T12:09:18.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.104+0000 7ff8f0ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:18.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.105+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:18.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.105+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request con 0x7ff8e0007c60 auth_method 0 2023-10-30T12:09:18.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.105+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:18.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.105+0000 7ff8f37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:18.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.105+0000 7ff8f37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:18.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.108+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:09:18.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.108+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:18.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.108+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:18.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.108+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:18.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.309+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:18.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.310+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request con 0x7ff8e0007c60 auth_method 0 2023-10-30T12:09:18.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.310+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:18.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.310+0000 7ff8f37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:18.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.310+0000 7ff8f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:18.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.310+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:09:18.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.310+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:18.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.310+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:18.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.310+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:18.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.744+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:18.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.772+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request con 0x7ff8e0007c60 auth_method 0 2023-10-30T12:09:18.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.772+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:18.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.772+0000 7ff8f37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:18.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.772+0000 7ff8f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:18.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.854+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:09:18.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.854+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:18.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.854+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:18.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:18.854+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:19.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.064+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139849310 0 0) 0x7ff8e8009530 con 0x7ff8e0009760 2023-10-30T12:09:19.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.064+0000 7ff8f1ffb640 10 monclient(hunting): _negotiate 2023-10-30T12:09:19.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.064+0000 7ff8f1ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:19.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.064+0000 7ff8f1ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:19.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.064+0000 7ff8f1ffb640 10 monclient(hunting): my global_id is 4099 2023-10-30T12:09:19.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.064+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8dc003520 con 0x7ff8e0009760 2023-10-30T12:09:19.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.065+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544436086 0 0) 0x7ff8e400f040 con 0x7ff8e000a6c0 2023-10-30T12:09:19.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.065+0000 7ff8f1ffb640 10 monclient(hunting): _negotiate 2023-10-30T12:09:19.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.065+0000 7ff8f1ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:19.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.065+0000 7ff8f1ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:19.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.065+0000 7ff8f1ffb640 10 monclient(hunting): my global_id is 4104 2023-10-30T12:09:19.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.065+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8dc003840 con 0x7ff8e000a6c0 2023-10-30T12:09:19.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2249146696 0 0) 0x7ff8e8009810 con 0x7ff8e0009760 2023-10-30T12:09:19.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 msgr2=0x7ff8e0008070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:19.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 1 --2- 172.21.15.3:0/2168404818 >> v2:172.21.15.92:3300/0 conn(0x7ff8e0007c60 0x7ff8e0008070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:19.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 >> v1:172.21.15.3:6789/0 conn(0x7ff8e000a6c0 legacy=0x7ff8e0009f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:19.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 10 monclient: _finish_hunting 0 2023-10-30T12:09:19.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:09:19.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:19.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.92:6789/0 2023-10-30T12:09:19.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f40aa360 con 0x7ff8e0009760 2023-10-30T12:09:19.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:09:19.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:09:19.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:19.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1777801622 0 0) 0x7ff8e8009bb0 con 0x7ff8e0009760 2023-10-30T12:09:19.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f9965640 5 monclient: authenticate success, global_id 4099 2023-10-30T12:09:19.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f9965640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:19.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:19.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:09:19.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 10 monclient: dump: 2023-10-30T12:09:19.169 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:09:19.169 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:19.169 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:19.169 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:19.169 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:19.170 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:19.170 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:19.170 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:19.170 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:19.170 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:09:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.162+0000 7ff8f1ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] but i'm connected to v1:172.21.15.92:6789/0, reconnecting 2023-10-30T12:09:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 >> v1:172.21.15.92:6789/0 conn(0x7ff8e0009760 legacy=0x7ff8e0013b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:09:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8dc003c90 0x7ff8dc004090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 10 monclient(hunting): picked mon.c con 0x7ff8dc003c90 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8dc0047c0 0x7ff8dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 10 monclient(hunting): picked mon.a con 0x7ff8dc0047c0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8dc005340 0x7ff8dc00b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 10 monclient(hunting): picked mon.b con 0x7ff8dc005340 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f4067a10 con 0x7ff8dc0047c0 2023-10-30T12:09:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc00c280 con 0x7ff8dc005340 2023-10-30T12:09:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc00c450 con 0x7ff8dc003c90 2023-10-30T12:09:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f1ffb640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:09:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f9965640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f2ffd640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8dc003c90 0x7ff8dc004090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request con 0x7ff8dc003c90 auth_method 0 2023-10-30T12:09:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.163+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.170+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.170+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.170+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:09:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.172+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8dc005340 0x7ff8dc00b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.173+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request con 0x7ff8dc005340 auth_method 0 2023-10-30T12:09:19.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.173+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:19.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.173+0000 7ff8f37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:19.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.173+0000 7ff8f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:19.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.175+0000 7ff8f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:19.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.175+0000 7ff8f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:19.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.175+0000 7ff8f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:09:19.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.185+0000 7ff8f37fe640 10 monclient(hunting): handle_auth_done global_id 4099 payload 293 2023-10-30T12:09:19.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.185+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8dc003c90 msgr2=0x7ff8dc004090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:19.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.185+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8dc003c90 0x7ff8dc004090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:19.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.185+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8dc0047c0 msgr2=0x7ff8dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:19.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.185+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8dc0047c0 0x7ff8dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:19.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.185+0000 7ff8f37fe640 10 monclient: _finish_hunting 0 2023-10-30T12:09:19.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.185+0000 7ff8f37fe640 1 monclient: found mon.b 2023-10-30T12:09:19.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.185+0000 7ff8f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:19.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.185+0000 7ff8f37fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:19.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.185+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e8009810 con 0x7ff8dc005340 2023-10-30T12:09:19.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.185+0000 7ff8f37fe640 10 monclient: _finish_auth 0 2023-10-30T12:09:19.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.185+0000 7ff8f37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:09:19.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.185+0000 7ff8f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:19.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.186+0000 7ff8f37fe640 1 -- 172.21.15.3:0/2168404818 reap_dead start 2023-10-30T12:09:19.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.188+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8dc005340 0x7ff8dc00b9a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009f30 tx=0x7ff8dc0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5bbcf2b034f9a9f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:19.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.219+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec031070 con 0x7ff8dc005340 2023-10-30T12:09:19.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.219+0000 7ff8f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:19.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.219+0000 7ff8f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:19.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.219+0000 7ff8f1ffb640 10 monclient: dump: 2023-10-30T12:09:19.222 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:09:19.222 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:19.222 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:19.223 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:19.223 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:19.223 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:19.223 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:19.223 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:19.224 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:19.224 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:09:19.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.219+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ec02da50 con 0x7ff8dc005340 2023-10-30T12:09:19.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.219+0000 7ff8f9965640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:19.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.219+0000 7ff8f1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:19.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.219+0000 7ff8f9965640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:19.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.219+0000 7ff8f1ffb640 1 -- 172.21.15.3:0/2168404818 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec02de10 con 0x7ff8dc005340 2023-10-30T12:09:19.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.219+0000 7ff8f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:19.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.219+0000 7ff8f9965640 4 set_mon_vals no callback set 2023-10-30T12:09:19.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.219+0000 7ff8f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:19.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.219+0000 7ff8f1ffb640 10 monclient: dump: 2023-10-30T12:09:19.226 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:09:19.226 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:19.226 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:19.227 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:19.227 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:19.227 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:19.227 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:19.227 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:19.228 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:19.228 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:09:19.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.220+0000 7ff8f9965640 10 monclient: shutdown 2023-10-30T12:09:19.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.220+0000 7ff8f9965640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:19.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.220+0000 7ff8f9965640 1 -- 172.21.15.3:0/2168404818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8dc005340 msgr2=0x7ff8dc00b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:19.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.220+0000 7ff8f9965640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8dc005340 0x7ff8dc00b9a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009f30 tx=0x7ff8dc0057f0 comp rx=0 tx=0).stop 2023-10-30T12:09:19.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.220+0000 7ff8f9965640 1 -- 172.21.15.3:0/2168404818 shutdown_connections 2023-10-30T12:09:19.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.220+0000 7ff8f9965640 1 --2- 172.21.15.3:0/2168404818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8dc005340 0x7ff8dc00b9a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:19.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.220+0000 7ff8f9965640 1 -- 172.21.15.3:0/2168404818 >> 172.21.15.3:0/2168404818 conn(0x7ff8f4065ea0 msgr2=0x7ff8f40662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:19.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.220+0000 7ff8f9965640 1 -- 172.21.15.3:0/2168404818 shutdown_connections 2023-10-30T12:09:19.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.220+0000 7ff8f9965640 1 -- 172.21.15.3:0/2168404818 wait complete. 2023-10-30T12:09:19.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.221+0000 7ff8f9965640 10 monclient: build_initial_monmap 2023-10-30T12:09:19.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.221+0000 7ff8f9965640 10 monclient: monmap: 2023-10-30T12:09:19.232 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:09:19.233 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:19.233 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:09:19.233 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:09:19.233 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:19.233 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:09:19.234 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:09:19.234 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:09:19.234 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:09:19.234 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:09:19.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 1 librados: starting msgr at 2023-10-30T12:09:19.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 1 librados: starting objecter 2023-10-30T12:09:19.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 1 Processor -- start 2023-10-30T12:09:19.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 1 -- start start 2023-10-30T12:09:19.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 1 librados: setting wanted keys 2023-10-30T12:09:19.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 1 librados: calling monclient init 2023-10-30T12:09:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 10 monclient: init 2023-10-30T12:09:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:09:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8dc005340 0x7ff8f41d9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 10 monclient(hunting): picked mon.noname-c con 0x7ff8dc005340 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:19.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f413f7a0 0x7ff8f4142b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:19.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 10 monclient(hunting): picked mon.noname-a con 0x7ff8f413f7a0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:19.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8f4141c40 0x7ff8f413b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:19.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 10 monclient(hunting): picked mon.noname-b con 0x7ff8f4141c40 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:19.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:19.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f40a8cb0 con 0x7ff8f413f7a0 2023-10-30T12:09:19.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:19.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f413fbe0 con 0x7ff8f4141c40 2023-10-30T12:09:19.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:19.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f41daad0 con 0x7ff8dc005340 2023-10-30T12:09:19.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:19.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.222+0000 7ff8f9965640 10 monclient(hunting): authenticate will time out at 1350.437056s 2023-10-30T12:09:19.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.231+0000 7ff8f27fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f413f7a0 0x7ff8f4142b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:19.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.235+0000 7ff8f27fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f413f7a0 0x7ff8f4142b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54502/0 (socket says 172.21.15.3:54502) 2023-10-30T12:09:19.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.235+0000 7ff8f27fc640 1 -- 172.21.15.3:0/3956725397 learned_addr learned my addr 172.21.15.3:0/3956725397 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:09:19.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.235+0000 7ff8f27fc640 10 monclient(hunting): get_auth_request con 0x7ff8f413f7a0 auth_method 0 2023-10-30T12:09:19.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.235+0000 7ff8f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:19.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.235+0000 7ff8f27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:19.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.235+0000 7ff8f27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:19.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.261+0000 7ff8f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:19.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.261+0000 7ff8f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:19.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.261+0000 7ff8f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:19.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.262+0000 7ff8f37fe640 1 --2- 172.21.15.3:0/3956725397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8f4141c40 0x7ff8f413b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:19.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.263+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request con 0x7ff8f4141c40 auth_method 0 2023-10-30T12:09:19.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.263+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:19.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.263+0000 7ff8f37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:19.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.263+0000 7ff8f37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:19.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.263+0000 7ff8f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:19.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.263+0000 7ff8f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:19.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.263+0000 7ff8f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:19.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.274+0000 7ff8f27fc640 10 monclient(hunting): handle_auth_done global_id 4107 payload 824 2023-10-30T12:09:19.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.274+0000 7ff8f27fc640 1 -- 172.21.15.3:0/3956725397 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8dc005340 msgr2=0x7ff8f41d9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:19.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.274+0000 7ff8f27fc640 1 --2- 172.21.15.3:0/3956725397 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8dc005340 0x7ff8f41d9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:19.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.274+0000 7ff8f27fc640 1 -- 172.21.15.3:0/3956725397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8f4141c40 msgr2=0x7ff8f413b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:19.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.274+0000 7ff8f27fc640 1 --2- 172.21.15.3:0/3956725397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8f4141c40 0x7ff8f413b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:19.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.274+0000 7ff8f27fc640 10 monclient: _finish_hunting 0 2023-10-30T12:09:19.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.274+0000 7ff8f27fc640 1 monclient: found mon.noname-a 2023-10-30T12:09:19.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.274+0000 7ff8f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:19.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.274+0000 7ff8f27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:09:19.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.274+0000 7ff8f27fc640 1 -- 172.21.15.3:0/3956725397 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f41430d0 con 0x7ff8f413f7a0 2023-10-30T12:09:19.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.281+0000 7ff8f27fc640 1 --2- 172.21.15.3:0/3956725397 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f413f7a0 0x7ff8f4142b90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8005ae0 tx=0x7ff8e8005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:19.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.281+0000 7ff8dbfff640 10 client.?.objecter ms_handle_connect 0x7ff8f413f7a0 2023-10-30T12:09:19.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.281+0000 7ff8dbfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:19.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.292+0000 7ff8dbfff640 1 -- 172.21.15.3:0/3956725397 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e800eb40 con 0x7ff8f413f7a0 2023-10-30T12:09:19.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.292+0000 7ff8dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:19.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.292+0000 7ff8dbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:19.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.292+0000 7ff8dbfff640 10 monclient: dump: 2023-10-30T12:09:19.296 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:09:19.296 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:19.296 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:19.296 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:19.297 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:19.297 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:19.297 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:19.297 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:19.297 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:19.298 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:09:19.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.292+0000 7ff8dbfff640 10 monclient: _finish_auth 0 2023-10-30T12:09:19.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.292+0000 7ff8dbfff640 10 monclient: _check_auth_tickets 2023-10-30T12:09:19.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.292+0000 7ff8dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:19.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8dbfff640 1 -- 172.21.15.3:0/3956725397 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e800ecb0 con 0x7ff8f413f7a0 2023-10-30T12:09:19.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8f9965640 5 monclient: authenticate success, global_id 4107 2023-10-30T12:09:19.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8f9965640 10 monclient: _renew_subs 2023-10-30T12:09:19.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8f9965640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:09:19.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8f9965640 1 -- 172.21.15.3:0/3956725397 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f41420c0 con 0x7ff8f413f7a0 2023-10-30T12:09:19.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8dbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:19.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8dbfff640 1 -- 172.21.15.3:0/3956725397 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e800c660 con 0x7ff8f413f7a0 2023-10-30T12:09:19.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:19.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:19.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8dbfff640 10 monclient: dump: 2023-10-30T12:09:19.301 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:09:19.301 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:19.301 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:19.301 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:19.301 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:19.303 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:09:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8f17fa640 4 set_mon_vals no callback set 2023-10-30T12:09:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8f9965640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8f9965640 10 monclient: _renew_subs 2023-10-30T12:09:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8f9965640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:09:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8f9965640 1 -- 172.21.15.3:0/3956725397 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f413b980 con 0x7ff8f413f7a0 2023-10-30T12:09:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8f9965640 1 librados: init done 2023-10-30T12:09:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.293+0000 7ff8f9965640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.294+0000 7ff8f9965640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.294+0000 7ff8f9965640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:09:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.294+0000 7ff8f9965640 1 -- 172.21.15.3:0/3956725397 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f404e9b0 con 0x7ff8f413f7a0 2023-10-30T12:09:19.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.532+0000 7ff8dbfff640 1 -- 172.21.15.3:0/3956725397 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff8e800cad0 con 0x7ff8f413f7a0 2023-10-30T12:09:19.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.532+0000 7ff8dbfff640 1 -- 172.21.15.3:0/3956725397 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff8e8025080 con 0x7ff8f413f7a0 2023-10-30T12:09:19.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.532+0000 7ff8dbfff640 10 client.4107.objecter ms_dispatch 0x7ff8f4000b90 osd_map(1..1 src has 1..1) v4 2023-10-30T12:09:19.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.532+0000 7ff8dbfff640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2023-10-30T12:09:19.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.532+0000 7ff8dbfff640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2023-10-30T12:09:19.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.532+0000 7ff8dbfff640 20 client.4107.objecter dump_active .. 0 homeless 2023-10-30T12:09:19.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.532+0000 7ff8dbfff640 1 -- 172.21.15.3:0/3956725397 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7ff8e800b430 con 0x7ff8f413f7a0 2023-10-30T12:09:19.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.532+0000 7ff8dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:19.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.532+0000 7ff8dbfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:19.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.602+0000 7ff8f9965640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-10-30T12:09:19.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.602+0000 7ff8f9965640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-10-30T12:09:19.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.602+0000 7ff8f9965640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:09:19.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:19.602+0000 7ff8f9965640 1 -- 172.21.15.3:0/3956725397 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff8f4142450 con 0x7ff8f413f7a0 2023-10-30T12:09:19.824 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:19.823+0000 7f0d0b844040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T12:09:19.857 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:19.856+0000 7f67faa3e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T12:09:19.883 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:19.882+0000 7f0d0b844040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T12:09:19.908 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:19.907+0000 7f67faa3e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T12:09:20.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.237+0000 7ff8dbfff640 1 -- 172.21.15.3:0/3956725397 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff8e800ee30 con 0x7ff8f413f7a0 2023-10-30T12:09:20.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.237+0000 7ff8dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-10-30T12:09:20.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.237+0000 7ff8dbfff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-10-30T12:09:20.240 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2023-10-30T12:09:20.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 10 librados: watch_flush enter 2023-10-30T12:09:20.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 10 librados: watch_flush exit 2023-10-30T12:09:20.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 20 client.4107.objecter shutdown clearing up homeless session... 2023-10-30T12:09:20.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 10 client.4107.objecter successfully canceled tick 2023-10-30T12:09:20.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 10 monclient: shutdown 2023-10-30T12:09:20.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:20.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 1 -- 172.21.15.3:0/3956725397 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f413f7a0 msgr2=0x7ff8f4142b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:20.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 1 --2- 172.21.15.3:0/3956725397 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f413f7a0 0x7ff8f4142b90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8005ae0 tx=0x7ff8e8005bd0 comp rx=0 tx=0).stop 2023-10-30T12:09:20.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 1 -- 172.21.15.3:0/3956725397 shutdown_connections 2023-10-30T12:09:20.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 1 --2- 172.21.15.3:0/3956725397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8f4141c40 0x7ff8f413b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:20.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 1 --2- 172.21.15.3:0/3956725397 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f413f7a0 0x7ff8f4142b90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:20.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 1 --2- 172.21.15.3:0/3956725397 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8dc005340 0x7ff8f41d9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:20.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 1 -- 172.21.15.3:0/3956725397 >> 172.21.15.3:0/3956725397 conn(0x7ff8f4065ea0 msgr2=0x7ff8f413dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:20.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 1 -- 172.21.15.3:0/3956725397 shutdown_connections 2023-10-30T12:09:20.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 1 -- 172.21.15.3:0/3956725397 wait complete. 2023-10-30T12:09:20.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:20.238+0000 7ff8f9965640 1 librados: shutdown 2023-10-30T12:09:20.251 INFO:tasks.ceph:check_enable_crimson: False 2023-10-30T12:09:20.293 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-30T12:09:20.294 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:09:20.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-30T12:09:20.327 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:09:20.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-30T12:09:20.401 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:09:20.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-30T12:09:20.430 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:20.429+0000 7f67faa3e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T12:09:20.472 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:09:20.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-30T12:09:20.474 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:20.472+0000 7f0d0b844040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T12:09:20.482 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:20.481+0000 7f67faa3e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T12:09:20.504 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:09:20.504 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-30T12:09:20.527 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:20.526+0000 7f67faa3e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T12:09:20.528 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:20.527+0000 7f0d0b844040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T12:09:20.531 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:09:20.532 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-30T12:09:20.575 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:20.574+0000 7f0d0b844040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T12:09:20.583 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:20.582+0000 7f67faa3e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T12:09:20.599 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:09:20.600 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-30T12:09:20.627 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:20.626+0000 7f67faa3e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T12:09:20.631 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:20.630+0000 7f0d0b844040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T12:09:20.654 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-10-30T12:09:20.654 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-30T12:09:20.674 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:20.673+0000 7f0d0b844040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T12:09:20.720 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 69005df2-28b6-4bb2-b3e6-79df0d403c70 0 2023-10-30T12:09:20.732 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:20.731+0000 7f67faa3e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-30T12:09:20.779 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:20.778+0000 7f0d0b844040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-30T12:09:20.806 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:20.804+0000 7f67faa3e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T12:09:20.829 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.828+0000 7ff9cf505640 10 monclient: get_monmap_and_config 2023-10-30T12:09:20.829 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.828+0000 7ff9cf505640 10 monclient: build_initial_monmap 2023-10-30T12:09:20.830 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.828+0000 7ff9cf505640 10 monclient: monmap: 2023-10-30T12:09:20.830 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:20.830 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:20.830 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:09:20.829031+0000 2023-10-30T12:09:20.831 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:09:20.829031+0000 2023-10-30T12:09:20.831 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:20.831 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:20.831 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:09:20.831 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:09:20.832 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:09:20.832 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:09:20.832 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:09:20.832 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:20.835 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.828+0000 7ff9cf505640 1 Processor -- start 2023-10-30T12:09:20.835 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.828+0000 7ff9cf505640 1 -- start start 2023-10-30T12:09:20.835 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.829+0000 7ff9cf505640 10 monclient: init 2023-10-30T12:09:20.836 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.829+0000 7ff9cf505640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:20.836 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.829+0000 7ff9cf505640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2023-10-30T12:09:20.836 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.829+0000 7ff9cf505640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff9c80b4440 0x7ff9c8144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:20.837 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.829+0000 7ff9cf505640 10 monclient(hunting): picked mon.noname-a con 0x7ff9c80b4440 addr v2:172.21.15.3:3300/0 2023-10-30T12:09:20.837 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.829+0000 7ff9cf505640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9c81453e0 addr v1:172.21.15.3:6789/0 2023-10-30T12:09:20.837 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.829+0000 7ff9cf505640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9c80b2230 0x7ff9c814c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:20.837 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.829+0000 7ff9cf505640 10 monclient(hunting): picked mon.noname-b con 0x7ff9c80b2230 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:20.838 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.829+0000 7ff9cf505640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c80a9da0 con 0x7ff9c81453e0 2023-10-30T12:09:20.838 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.829+0000 7ff9cf505640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:20.838 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.829+0000 7ff9cf505640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9c8067a10 con 0x7ff9c80b4440 2023-10-30T12:09:20.838 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.829+0000 7ff9cf505640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:20.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.829+0000 7ff9cf505640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c80aa080 con 0x7ff9c80b2230 2023-10-30T12:09:20.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.829+0000 7ff9cf505640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:20.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.829+0000 7ff9cf505640 10 monclient(hunting): authenticate will time out at 1358.006923s 2023-10-30T12:09:20.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.830+0000 7ff9cd2a1640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff9c80b4440 0x7ff9c8144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:20.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.830+0000 7ff9cdaa2640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9c80b2230 0x7ff9c814c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:20.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.830+0000 7ff9cdaa2640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9c80b2230 0x7ff9c814c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.92:39626/0 (socket says 172.21.15.92:39626) 2023-10-30T12:09:20.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.830+0000 7ff9cdaa2640 1 -- 172.21.15.92:0/114812506 learned_addr learned my addr 172.21.15.92:0/114812506 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-10-30T12:09:20.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.830+0000 7ff9cd2a1640 10 monclient(hunting): get_auth_request con 0x7ff9c80b4440 auth_method 0 2023-10-30T12:09:20.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.830+0000 7ff9cd2a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:20.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.830+0000 7ff9cd2a1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:20.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.830+0000 7ff9cd2a1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:20.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.830+0000 7ff9cdaa2640 10 monclient(hunting): get_auth_request con 0x7ff9c80b2230 auth_method 0 2023-10-30T12:09:20.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.830+0000 7ff9cdaa2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:20.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.830+0000 7ff9cdaa2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:20.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.830+0000 7ff9cdaa2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:20.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.831+0000 7ff9cdaa2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:20.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.831+0000 7ff9cdaa2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:20.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.831+0000 7ff9cdaa2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:20.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.831+0000 7ff9cd2a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:20.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.831+0000 7ff9cd2a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:20.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.831+0000 7ff9cd2a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:20.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.833+0000 7ff9cdaa2640 10 monclient(hunting): handle_auth_done global_id 4106 payload 274 2023-10-30T12:09:20.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.833+0000 7ff9cdaa2640 1 -- 172.21.15.92:0/114812506 >> v2:172.21.15.3:3300/0 conn(0x7ff9c80b4440 msgr2=0x7ff9c8144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:20.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.833+0000 7ff9cdaa2640 1 --2- 172.21.15.92:0/114812506 >> v2:172.21.15.3:3300/0 conn(0x7ff9c80b4440 0x7ff9c8144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:20.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.833+0000 7ff9cdaa2640 1 -- 172.21.15.92:0/114812506 >> v1:172.21.15.3:6789/0 conn(0x7ff9c81453e0 legacy=0x7ff9c80b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:20.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.833+0000 7ff9cdaa2640 10 monclient: _finish_hunting 0 2023-10-30T12:09:20.844 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.833+0000 7ff9cdaa2640 1 monclient: found mon.noname-b 2023-10-30T12:09:20.844 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.833+0000 7ff9cdaa2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:20.844 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.833+0000 7ff9cdaa2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:09:20.844 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.833+0000 7ff9cdaa2640 1 -- 172.21.15.92:0/114812506 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c80aa360 con 0x7ff9c80b2230 2023-10-30T12:09:20.849 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.835+0000 7ff9cdaa2640 1 --2- 172.21.15.92:0/114812506 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9c80b2230 0x7ff9c814c3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009e50 tx=0x7ff9c002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b8fa77e5a3e80c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:20.849 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9bffff640 1 -- 172.21.15.92:0/114812506 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c002c9e0 con 0x7ff9c80b2230 2023-10-30T12:09:20.849 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:20.849 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9bffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:20.850 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9bffff640 10 monclient: dump: 2023-10-30T12:09:20.850 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:20.850 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:20.850 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:20.850 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:20.851 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:20.851 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:20.851 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:20.851 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:20.851 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:20.851 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:20.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9bffff640 10 monclient: _finish_auth 0 2023-10-30T12:09:20.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9bffff640 10 monclient: _check_auth_tickets 2023-10-30T12:09:20.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:20.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9bffff640 1 -- 172.21.15.92:0/114812506 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c002cb50 con 0x7ff9c80b2230 2023-10-30T12:09:20.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9cf505640 5 monclient: authenticate success, global_id 4106 2023-10-30T12:09:20.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9cf505640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:20.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9bffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:20.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9cf505640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:20.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9bffff640 1 -- 172.21.15.92:0/114812506 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c00343f0 con 0x7ff9c80b2230 2023-10-30T12:09:20.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:20.854 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9cf505640 4 set_mon_vals no callback set 2023-10-30T12:09:20.854 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:20.854 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9bffff640 10 monclient: dump: 2023-10-30T12:09:20.854 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:20.854 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:20.855 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:20.855 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:20.855 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:20.855 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:20.855 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:20.856 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:20.856 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:20.856 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:20.856 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9cf505640 10 monclient: shutdown 2023-10-30T12:09:20.856 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9cf505640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:20.857 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9cf505640 1 -- 172.21.15.92:0/114812506 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9c80b2230 msgr2=0x7ff9c814c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:20.857 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9cf505640 1 --2- 172.21.15.92:0/114812506 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9c80b2230 0x7ff9c814c3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009e50 tx=0x7ff9c002be40 comp rx=0 tx=0).stop 2023-10-30T12:09:20.857 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9cf505640 1 -- 172.21.15.92:0/114812506 shutdown_connections 2023-10-30T12:09:20.857 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9cf505640 1 --2- 172.21.15.92:0/114812506 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9c80b2230 0x7ff9c814c3e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:20.857 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9cf505640 1 --2- 172.21.15.92:0/114812506 >> v2:172.21.15.3:3300/0 conn(0x7ff9c80b4440 0x7ff9c8144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:20.857 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.841+0000 7ff9cf505640 1 -- 172.21.15.92:0/114812506 >> 172.21.15.92:0/114812506 conn(0x7ff9c8065ea0 msgr2=0x7ff9c80662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:20.858 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.842+0000 7ff9cf505640 1 -- 172.21.15.92:0/114812506 shutdown_connections 2023-10-30T12:09:20.858 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.842+0000 7ff9cf505640 1 -- 172.21.15.92:0/114812506 wait complete. 2023-10-30T12:09:20.858 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.842+0000 7ff9cf505640 10 monclient: build_initial_monmap 2023-10-30T12:09:20.858 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.842+0000 7ff9cf505640 10 monclient: monmap: 2023-10-30T12:09:20.859 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:20.859 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:20.859 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-10-30T12:09:20.859 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-10-30T12:09:20.859 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:20.859 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-10-30T12:09:20.860 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:09:20.860 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:09:20.860 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:09:20.860 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:20.860 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 1 librados: starting msgr at 2023-10-30T12:09:20.861 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 1 librados: starting objecter 2023-10-30T12:09:20.861 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 1 Processor -- start 2023-10-30T12:09:20.861 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 1 -- start start 2023-10-30T12:09:20.861 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 1 librados: setting wanted keys 2023-10-30T12:09:20.861 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 1 librados: calling monclient init 2023-10-30T12:09:20.861 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 10 monclient: init 2023-10-30T12:09:20.862 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:20.862 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:09:20.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c80b2230 0x7ff9c80b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:20.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 10 monclient(hunting): picked mon.noname-b con 0x7ff9c80b2230 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:20.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9c80b4440 0x7ff9c81cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:20.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 10 monclient(hunting): picked mon.noname-a con 0x7ff9c80b4440 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:20.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9c81453e0 0x7ff9c81ca710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:20.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 10 monclient(hunting): picked mon.noname-c con 0x7ff9c81453e0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:20.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:20.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c80a8b50 con 0x7ff9c80b4440 2023-10-30T12:09:20.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:20.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c81cc9a0 con 0x7ff9c80b2230 2023-10-30T12:09:20.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:20.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c813fc90 con 0x7ff9c81453e0 2023-10-30T12:09:20.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:20.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.843+0000 7ff9cf505640 10 monclient(hunting): authenticate will time out at 1358.021220s 2023-10-30T12:09:20.866 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.844+0000 7ff9cd2a1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c80b2230 0x7ff9c80b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:20.866 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.844+0000 7ff9ccaa0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9c80b4440 0x7ff9c81cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:20.866 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.844+0000 7ff9cd2a1640 1 -- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c80b2230 msgr2=0x7ff9c80b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:09:20.866 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.844+0000 7ff9cd2a1640 1 -- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c80b2230 msgr2=0x7ff9c80b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:09:20.866 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.844+0000 7ff9cd2a1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c80b2230 0x7ff9c80b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:20.867 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.844+0000 7ff9cd2a1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c80b2230 0x7ff9c80b0b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:20.867 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.845+0000 7ff9ccaa0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9c80b4440 0x7ff9c81cd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.92:36838/0 (socket says 172.21.15.92:36838) 2023-10-30T12:09:20.867 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.845+0000 7ff9ccaa0640 1 -- 172.21.15.92:0/1691575637 learned_addr learned my addr 172.21.15.92:0/1691575637 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-10-30T12:09:20.867 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.845+0000 7ff9ccaa0640 10 monclient(hunting): get_auth_request con 0x7ff9c80b4440 auth_method 0 2023-10-30T12:09:20.867 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.845+0000 7ff9ccaa0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:20.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.845+0000 7ff9ccaa0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:20.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.845+0000 7ff9ccaa0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:20.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.845+0000 7ff9ccaa0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:20.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.845+0000 7ff9ccaa0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:20.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.845+0000 7ff9ccaa0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:20.869 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:20.854+0000 7f0d0b844040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T12:09:20.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.848+0000 7ff9ccaa0640 10 monclient(hunting): handle_auth_done global_id 4116 payload 824 2023-10-30T12:09:20.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.848+0000 7ff9ccaa0640 1 -- 172.21.15.92:0/1691575637 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9c81453e0 msgr2=0x7ff9c81ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:20.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.848+0000 7ff9ccaa0640 1 --2- 172.21.15.92:0/1691575637 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9c81453e0 0x7ff9c81ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:20.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.848+0000 7ff9ccaa0640 1 -- 172.21.15.92:0/1691575637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c80b2230 msgr2=0x7ff9c80b0b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:09:20.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.848+0000 7ff9ccaa0640 1 --2- 172.21.15.92:0/1691575637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c80b2230 0x7ff9c80b0b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:20.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.848+0000 7ff9ccaa0640 10 monclient: _finish_hunting 0 2023-10-30T12:09:20.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.848+0000 7ff9ccaa0640 1 monclient: found mon.noname-a 2023-10-30T12:09:20.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.848+0000 7ff9ccaa0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:20.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.848+0000 7ff9ccaa0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:09:20.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.848+0000 7ff9ccaa0640 1 -- 172.21.15.92:0/1691575637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c81cb7f0 con 0x7ff9c80b4440 2023-10-30T12:09:20.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.850+0000 7ff9ccaa0640 1 --2- 172.21.15.92:0/1691575637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9c80b4440 0x7ff9c81cd0e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400c400 tx=0x7ff9c400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:20.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.851+0000 7ff9bdffb640 10 client.?.objecter ms_handle_connect 0x7ff9c80b4440 2023-10-30T12:09:20.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.851+0000 7ff9bdffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:20.873 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:20.853+0000 7f67faa3e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T12:09:20.874 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9bdffb640 1 -- 172.21.15.92:0/1691575637 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c40052d0 con 0x7ff9c80b4440 2023-10-30T12:09:20.874 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:20.874 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9bdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:20.875 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9bdffb640 10 monclient: dump: 2023-10-30T12:09:20.875 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:20.875 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:20.875 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:20.876 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:20.876 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:20.876 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:20.876 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:20.876 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:20.876 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:20.877 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:20.877 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9bdffb640 10 monclient: _finish_auth 0 2023-10-30T12:09:20.877 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9bdffb640 10 monclient: _check_auth_tickets 2023-10-30T12:09:20.877 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:20.878 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9cf505640 5 monclient: authenticate success, global_id 4116 2023-10-30T12:09:20.878 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9bdffb640 1 -- 172.21.15.92:0/1691575637 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c400e040 con 0x7ff9c80b4440 2023-10-30T12:09:20.878 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9cf505640 10 monclient: _renew_subs 2023-10-30T12:09:20.878 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9cf505640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:09:20.878 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9cf505640 1 -- 172.21.15.92:0/1691575637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c81ce490 con 0x7ff9c80b4440 2023-10-30T12:09:20.879 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9bdffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:20.879 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9bdffb640 1 -- 172.21.15.92:0/1691575637 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c40035f0 con 0x7ff9c80b4440 2023-10-30T12:09:20.879 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:20.879 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:20.879 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9bdffb640 10 monclient: dump: 2023-10-30T12:09:20.880 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:20.880 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:20.880 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:20.880 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:20.880 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:20.881 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:20.881 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:20.881 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:20.881 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:20.881 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:20.881 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9bf7fe640 4 set_mon_vals no callback set 2023-10-30T12:09:20.882 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9cf505640 10 client.4116.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:20.882 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9cf505640 10 monclient: _renew_subs 2023-10-30T12:09:20.882 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9cf505640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:09:20.882 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9cf505640 1 -- 172.21.15.92:0/1691575637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c81d9dd0 con 0x7ff9c80b4440 2023-10-30T12:09:20.882 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.854+0000 7ff9cf505640 1 librados: init done 2023-10-30T12:09:20.883 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.855+0000 7ff9cf505640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:20.883 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.855+0000 7ff9cf505640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:20.883 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.855+0000 7ff9cf505640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:09:20.883 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.855+0000 7ff9cf505640 1 -- 172.21.15.92:0/1691575637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c804e9b0 con 0x7ff9c80b4440 2023-10-30T12:09:20.904 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:20.903+0000 7f0d0b844040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T12:09:20.906 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:20.905+0000 7f67faa3e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T12:09:20.953 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:20.950+0000 7f67faa3e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T12:09:20.958 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.957+0000 7ff9bdffb640 1 -- 172.21.15.92:0/1691575637 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff9c4003a60 con 0x7ff9c80b4440 2023-10-30T12:09:20.959 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:20.958+0000 7f0d0b844040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T12:09:20.963 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.962+0000 7ff9bdffb640 1 -- 172.21.15.92:0/1691575637 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff9c401c780 con 0x7ff9c80b4440 2023-10-30T12:09:20.963 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.962+0000 7ff9bdffb640 10 client.4116.objecter ms_dispatch 0x7ff9c8000b90 osd_map(2..2 src has 1..2) v4 2023-10-30T12:09:20.964 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.962+0000 7ff9bdffb640 3 client.4116.objecter handle_osd_map got epochs [2,2] > 0 2023-10-30T12:09:20.964 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.962+0000 7ff9bdffb640 3 client.4116.objecter handle_osd_map decoding full epoch 2 2023-10-30T12:09:20.964 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.962+0000 7ff9bdffb640 20 client.4116.objecter dump_active .. 0 homeless 2023-10-30T12:09:20.964 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.962+0000 7ff9bdffb640 1 -- 172.21.15.92:0/1691575637 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7ff9c4011cf0 con 0x7ff9c80b4440 2023-10-30T12:09:20.965 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.962+0000 7ff9bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:20.965 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:20.962+0000 7ff9bdffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:21.008 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:21.007+0000 7f0d0b844040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T12:09:21.032 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.031+0000 7ff9cf505640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "69005df2-28b6-4bb2-b3e6-79df0d403c70", "id": 0}] 2023-10-30T12:09:21.033 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.031+0000 7ff9cf505640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "69005df2-28b6-4bb2-b3e6-79df0d403c70", "id": 0}] 2023-10-30T12:09:21.033 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.031+0000 7ff9cf505640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:09:21.033 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.031+0000 7ff9cf505640 1 -- 172.21.15.92:0/1691575637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd new", "uuid": "69005df2-28b6-4bb2-b3e6-79df0d403c70", "id": 0} v 0) v1 -- 0x7ff9c8074370 con 0x7ff9c80b4440 2023-10-30T12:09:21.180 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.178+0000 7ff9bdffb640 1 -- 172.21.15.92:0/1691575637 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "69005df2-28b6-4bb2-b3e6-79df0d403c70", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff9c4011590 con 0x7ff9c80b4440 2023-10-30T12:09:21.180 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.178+0000 7ff9bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "69005df2-28b6-4bb2-b3e6-79df0d403c70", "id": 0}] 2023-10-30T12:09:21.180 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.178+0000 7ff9bdffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:21.181 INFO:teuthology.orchestra.run.smithi092.stdout:0 2023-10-30T12:09:21.182 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.179+0000 7ff9cf505640 10 librados: watch_flush enter 2023-10-30T12:09:21.182 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.179+0000 7ff9cf505640 10 librados: watch_flush exit 2023-10-30T12:09:21.182 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.179+0000 7ff9cf505640 20 client.4116.objecter shutdown clearing up homeless session... 2023-10-30T12:09:21.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.179+0000 7ff9cf505640 10 client.4116.objecter successfully canceled tick 2023-10-30T12:09:21.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.179+0000 7ff9cf505640 10 monclient: shutdown 2023-10-30T12:09:21.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.179+0000 7ff9cf505640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:21.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.179+0000 7ff9cf505640 1 -- 172.21.15.92:0/1691575637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9c80b4440 msgr2=0x7ff9c81cd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:21.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.179+0000 7ff9cf505640 1 --2- 172.21.15.92:0/1691575637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9c80b4440 0x7ff9c81cd0e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400c400 tx=0x7ff9c400c8c0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.180+0000 7ff9cf505640 1 -- 172.21.15.92:0/1691575637 shutdown_connections 2023-10-30T12:09:21.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.180+0000 7ff9cf505640 1 --2- 172.21.15.92:0/1691575637 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9c81453e0 0x7ff9c81ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.180+0000 7ff9cf505640 1 --2- 172.21.15.92:0/1691575637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9c80b4440 0x7ff9c81cd0e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.180+0000 7ff9cf505640 1 --2- 172.21.15.92:0/1691575637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c80b2230 0x7ff9c80b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.180+0000 7ff9cf505640 1 -- 172.21.15.92:0/1691575637 >> 172.21.15.92:0/1691575637 conn(0x7ff9c8065ea0 msgr2=0x7ff9c80b78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:21.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.180+0000 7ff9cf505640 1 -- 172.21.15.92:0/1691575637 shutdown_connections 2023-10-30T12:09:21.186 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.180+0000 7ff9cf505640 1 -- 172.21.15.92:0/1691575637 wait complete. 2023-10-30T12:09:21.186 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.180+0000 7ff9cf505640 1 librados: shutdown 2023-10-30T12:09:21.191 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 6466ea5c-d74a-4709-a83a-f5d4fe17157e 1 2023-10-30T12:09:21.296 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.295+0000 7feb3de21640 10 monclient: get_monmap_and_config 2023-10-30T12:09:21.297 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.295+0000 7feb3de21640 10 monclient: build_initial_monmap 2023-10-30T12:09:21.297 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.295+0000 7feb3de21640 10 monclient: monmap: 2023-10-30T12:09:21.297 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:21.297 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:21.298 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:09:21.296438+0000 2023-10-30T12:09:21.298 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:09:21.296438+0000 2023-10-30T12:09:21.298 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:21.298 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:21.298 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:09:21.299 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:09:21.299 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:09:21.299 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:09:21.300 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:09:21.300 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:21.301 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.296+0000 7feb3de21640 1 Processor -- start 2023-10-30T12:09:21.301 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.296+0000 7feb3de21640 1 -- start start 2023-10-30T12:09:21.301 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.296+0000 7feb3de21640 10 monclient: init 2023-10-30T12:09:21.302 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.296+0000 7feb3de21640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:21.302 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.296+0000 7feb3de21640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-10-30T12:09:21.302 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.296+0000 7feb3de21640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb380af870 addr v1:172.21.15.3:6789/0 2023-10-30T12:09:21.302 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.296+0000 7feb3de21640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feb380afc50 0x7feb3813f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:21.302 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.296+0000 7feb3de21640 10 monclient(hunting): picked mon.noname-a con 0x7feb380afc50 addr v2:172.21.15.3:3300/0 2023-10-30T12:09:21.303 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.296+0000 7feb3de21640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7feb38070ff0 0x7feb3813cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:21.303 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.296+0000 7feb3de21640 10 monclient(hunting): picked mon.noname-c con 0x7feb38070ff0 addr v2:172.21.15.92:3300/0 2023-10-30T12:09:21.303 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.296+0000 7feb3de21640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb380707a0 con 0x7feb380af870 2023-10-30T12:09:21.303 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.297+0000 7feb3de21640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:21.303 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.297+0000 7feb3de21640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb38074040 con 0x7feb380afc50 2023-10-30T12:09:21.304 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.297+0000 7feb3de21640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:21.304 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.297+0000 7feb3de21640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb3813fca0 con 0x7feb38070ff0 2023-10-30T12:09:21.304 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.297+0000 7feb3de21640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:21.304 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.297+0000 7feb3de21640 10 monclient(hunting): authenticate will time out at 1358.474262s 2023-10-30T12:09:21.307 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.306+0000 7feb37fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7feb38070ff0 0x7feb3813cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:21.308 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.307+0000 7feb37fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7feb38070ff0 0x7feb3813cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:34296/0 (socket says 172.21.15.92:34296) 2023-10-30T12:09:21.308 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.307+0000 7feb37fff640 1 -- 172.21.15.92:0/1579733933 learned_addr learned my addr 172.21.15.92:0/1579733933 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-10-30T12:09:21.309 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.307+0000 7feb37fff640 10 monclient(hunting): get_auth_request con 0x7feb38070ff0 auth_method 0 2023-10-30T12:09:21.309 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.307+0000 7feb37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:21.309 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.307+0000 7feb37fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:21.309 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.307+0000 7feb37fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:21.311 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.308+0000 7feb36ffd640 1 --2- 172.21.15.92:0/1579733933 >> v2:172.21.15.3:3300/0 conn(0x7feb380afc50 0x7feb3813f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:21.311 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.308+0000 7feb37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:21.311 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.308+0000 7feb37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:21.312 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.308+0000 7feb37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:21.312 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.309+0000 7feb36ffd640 10 monclient(hunting): get_auth_request con 0x7feb380afc50 auth_method 0 2023-10-30T12:09:21.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.309+0000 7feb36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:21.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.309+0000 7feb36ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:21.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.309+0000 7feb36ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:21.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.310+0000 7feb37fff640 10 monclient(hunting): handle_auth_done global_id 4108 payload 274 2023-10-30T12:09:21.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.310+0000 7feb37fff640 1 -- 172.21.15.92:0/1579733933 >> v2:172.21.15.3:3300/0 conn(0x7feb380afc50 msgr2=0x7feb3813f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:21.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.310+0000 7feb37fff640 1 --2- 172.21.15.92:0/1579733933 >> v2:172.21.15.3:3300/0 conn(0x7feb380afc50 0x7feb3813f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.310+0000 7feb37fff640 1 -- 172.21.15.92:0/1579733933 >> v1:172.21.15.3:6789/0 conn(0x7feb380af870 legacy=0x7feb380ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:21.316 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.310+0000 7feb37fff640 10 monclient: _finish_hunting 0 2023-10-30T12:09:21.316 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.310+0000 7feb37fff640 1 monclient: found mon.noname-c 2023-10-30T12:09:21.316 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.310+0000 7feb37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:21.317 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.310+0000 7feb37fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:09:21.317 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.310+0000 7feb37fff640 1 -- 172.21.15.92:0/1579733933 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb38071ee0 con 0x7feb38070ff0 2023-10-30T12:09:21.317 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.313+0000 7feb37fff640 1 --2- 172.21.15.92:0/1579733933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb38070ff0 0x7feb3813cd30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feb380a8c20 tx=0x7feb3002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39922372ec62d698 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:21.320 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.316+0000 7feb367fc640 1 -- 172.21.15.92:0/1579733933 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb300317e0 con 0x7feb38070ff0 2023-10-30T12:09:21.320 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:21.320 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:21.320 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb367fc640 10 monclient: dump: 2023-10-30T12:09:21.321 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:21.321 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:21.321 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.321 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.322 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:21.322 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:21.322 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:21.322 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:21.322 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:21.323 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:21.323 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb367fc640 10 monclient: _finish_auth 0 2023-10-30T12:09:21.323 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb367fc640 10 monclient: _check_auth_tickets 2023-10-30T12:09:21.323 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:21.323 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb367fc640 1 -- 172.21.15.92:0/1579733933 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb30031950 con 0x7feb38070ff0 2023-10-30T12:09:21.324 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb367fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:21.324 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb367fc640 1 -- 172.21.15.92:0/1579733933 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30031cf0 con 0x7feb38070ff0 2023-10-30T12:09:21.324 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb3de21640 5 monclient: authenticate success, global_id 4108 2023-10-30T12:09:21.324 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb3de21640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:21.324 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb3de21640 4 set_mon_vals no callback set 2023-10-30T12:09:21.325 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:21.325 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:21.325 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb367fc640 10 monclient: dump: 2023-10-30T12:09:21.325 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:21.325 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:21.325 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.326 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.326 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:21.326 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:21.326 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:21.326 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:21.327 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:21.327 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:21.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb3de21640 10 monclient: shutdown 2023-10-30T12:09:21.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb3de21640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:21.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb3de21640 1 -- 172.21.15.92:0/1579733933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb38070ff0 msgr2=0x7feb3813cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:21.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb3de21640 1 --2- 172.21.15.92:0/1579733933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb38070ff0 0x7feb3813cd30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feb380a8c20 tx=0x7feb3002b880 comp rx=0 tx=0).stop 2023-10-30T12:09:21.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb3de21640 1 -- 172.21.15.92:0/1579733933 shutdown_connections 2023-10-30T12:09:21.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb3de21640 1 --2- 172.21.15.92:0/1579733933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb38070ff0 0x7feb3813cd30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb3de21640 1 --2- 172.21.15.92:0/1579733933 >> v2:172.21.15.3:3300/0 conn(0x7feb380afc50 0x7feb3813f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb3de21640 1 -- 172.21.15.92:0/1579733933 >> 172.21.15.92:0/1579733933 conn(0x7feb380b0960 msgr2=0x7feb380b0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:21.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb3de21640 1 -- 172.21.15.92:0/1579733933 shutdown_connections 2023-10-30T12:09:21.329 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.317+0000 7feb3de21640 1 -- 172.21.15.92:0/1579733933 wait complete. 2023-10-30T12:09:21.329 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.318+0000 7feb3de21640 10 monclient: build_initial_monmap 2023-10-30T12:09:21.329 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.318+0000 7feb3de21640 10 monclient: monmap: 2023-10-30T12:09:21.329 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:21.330 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:21.330 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-10-30T12:09:21.330 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-10-30T12:09:21.330 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:21.330 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-10-30T12:09:21.331 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:09:21.331 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:09:21.331 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:09:21.331 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:21.331 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.318+0000 7feb3de21640 1 librados: starting msgr at 2023-10-30T12:09:21.332 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.318+0000 7feb3de21640 1 librados: starting objecter 2023-10-30T12:09:21.332 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.318+0000 7feb3de21640 1 Processor -- start 2023-10-30T12:09:21.332 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 1 -- start start 2023-10-30T12:09:21.332 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 1 librados: setting wanted keys 2023-10-30T12:09:21.332 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 1 librados: calling monclient init 2023-10-30T12:09:21.333 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 10 monclient: init 2023-10-30T12:09:21.337 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:21.337 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:09:21.337 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb38070ff0 0x7feb380ac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:21.337 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 10 monclient(hunting): picked mon.noname-c con 0x7feb38070ff0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:21.337 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb380af870 0x7feb380aca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:21.338 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 10 monclient(hunting): picked mon.noname-b con 0x7feb380af870 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:21.338 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb380a9f00 0x7feb380aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:21.338 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 10 monclient(hunting): picked mon.noname-a con 0x7feb380a9f00 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:21.338 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:21.338 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb380b0480 con 0x7feb380a9f00 2023-10-30T12:09:21.339 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:21.339 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb380abb60 con 0x7feb380af870 2023-10-30T12:09:21.339 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:21.339 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb380a9700 con 0x7feb38070ff0 2023-10-30T12:09:21.340 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:21.340 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.319+0000 7feb3de21640 10 monclient(hunting): authenticate will time out at 1358.496941s 2023-10-30T12:09:21.340 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb36ffd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb380af870 0x7feb380aca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:21.340 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb377fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb38070ff0 0x7feb380ac4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:21.340 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb37fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb380a9f00 0x7feb380aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:21.341 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb36ffd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb380af870 0x7feb380aca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:34302/0 (socket says 172.21.15.92:34302) 2023-10-30T12:09:21.341 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb36ffd640 1 -- 172.21.15.92:0/728345421 learned_addr learned my addr 172.21.15.92:0/728345421 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-10-30T12:09:21.341 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb36ffd640 10 monclient(hunting): get_auth_request con 0x7feb380af870 auth_method 0 2023-10-30T12:09:21.341 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:21.342 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb36ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:21.342 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb36ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:21.342 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb377fe640 10 monclient(hunting): get_auth_request con 0x7feb38070ff0 auth_method 0 2023-10-30T12:09:21.342 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:21.342 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:21.343 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb377fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:21.343 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb37fff640 10 monclient(hunting): get_auth_request con 0x7feb380a9f00 auth_method 0 2023-10-30T12:09:21.343 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:21.343 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb37fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:21.343 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.320+0000 7feb37fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:21.344 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.321+0000 7feb36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:21.344 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.321+0000 7feb36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:21.344 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.321+0000 7feb36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:21.344 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.321+0000 7feb377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:21.344 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.321+0000 7feb377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:21.345 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.321+0000 7feb377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:21.345 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.321+0000 7feb37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:21.345 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.321+0000 7feb37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:21.345 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.321+0000 7feb37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:21.345 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.323+0000 7feb36ffd640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2023-10-30T12:09:21.346 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.323+0000 7feb36ffd640 1 -- 172.21.15.92:0/728345421 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb38070ff0 msgr2=0x7feb380ac4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:21.346 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.323+0000 7feb36ffd640 1 --2- 172.21.15.92:0/728345421 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb38070ff0 0x7feb380ac4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.346 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.323+0000 7feb36ffd640 1 -- 172.21.15.92:0/728345421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb380a9f00 msgr2=0x7feb380aa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:21.346 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.323+0000 7feb36ffd640 1 --2- 172.21.15.92:0/728345421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb380a9f00 0x7feb380aa390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.346 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.323+0000 7feb36ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:09:21.347 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.323+0000 7feb36ffd640 1 monclient: found mon.noname-b 2023-10-30T12:09:21.347 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.323+0000 7feb36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:21.347 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.323+0000 7feb36ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:09:21.347 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.323+0000 7feb36ffd640 1 -- 172.21.15.92:0/728345421 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb380a99e0 con 0x7feb380af870 2023-10-30T12:09:21.347 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.325+0000 7feb36ffd640 1 --2- 172.21.15.92:0/728345421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb380af870 0x7feb380aca30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00d830 tx=0x7feb2c00dcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:21.348 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.325+0000 7feb1ffff640 10 client.?.objecter ms_handle_connect 0x7feb380af870 2023-10-30T12:09:21.348 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.325+0000 7feb1ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:21.348 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb1ffff640 1 -- 172.21.15.92:0/728345421 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c00b6c0 con 0x7feb380af870 2023-10-30T12:09:21.348 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:21.348 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb1ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:21.349 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb1ffff640 10 monclient: dump: 2023-10-30T12:09:21.349 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:21.349 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:21.349 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.349 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.349 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:21.350 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:21.350 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:21.350 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:21.350 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:21.351 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:21.351 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb1ffff640 10 monclient: _finish_auth 0 2023-10-30T12:09:21.351 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb1ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:09:21.351 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:21.352 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb1ffff640 1 -- 172.21.15.92:0/728345421 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c00b830 con 0x7feb380af870 2023-10-30T12:09:21.352 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb1ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:21.353 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb1ffff640 1 -- 172.21.15.92:0/728345421 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c00bbd0 con 0x7feb380af870 2023-10-30T12:09:21.353 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:21.353 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:21.354 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb1ffff640 10 monclient: dump: 2023-10-30T12:09:21.354 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:21.354 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:21.355 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.355 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.355 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:21.356 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:21.356 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:21.356 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:21.356 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:21.357 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:21.357 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb3de21640 5 monclient: authenticate success, global_id 4111 2023-10-30T12:09:21.357 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb3de21640 10 monclient: _renew_subs 2023-10-30T12:09:21.357 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb3de21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:21.358 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb35ffb640 4 set_mon_vals no callback set 2023-10-30T12:09:21.358 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb3de21640 1 -- 172.21.15.92:0/728345421 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb380a8ac0 con 0x7feb380af870 2023-10-30T12:09:21.359 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb3de21640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:21.359 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb3de21640 10 monclient: _renew_subs 2023-10-30T12:09:21.359 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb3de21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:21.359 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb3de21640 1 -- 172.21.15.92:0/728345421 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb380aaab0 con 0x7feb380af870 2023-10-30T12:09:21.359 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.330+0000 7feb3de21640 1 librados: init done 2023-10-30T12:09:21.360 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.331+0000 7feb3de21640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:21.360 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.331+0000 7feb3de21640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:21.360 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.331+0000 7feb3de21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:21.360 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.331+0000 7feb3de21640 1 -- 172.21.15.92:0/728345421 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb3804e9b0 con 0x7feb380af870 2023-10-30T12:09:21.361 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:21.345+0000 7f67faa3e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-30T12:09:21.422 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:21.420+0000 7f0d0b844040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-30T12:09:21.492 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:21.491+0000 7f67faa3e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T12:09:21.530 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.528+0000 7feb1ffff640 1 -- 172.21.15.92:0/728345421 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7feb2c01e760 con 0x7feb380af870 2023-10-30T12:09:21.530 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.528+0000 7feb1ffff640 1 -- 172.21.15.92:0/728345421 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7feb2c028ae0 con 0x7feb380af870 2023-10-30T12:09:21.531 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.528+0000 7feb1ffff640 10 client.4111.objecter ms_dispatch 0x7feb38000b90 osd_map(3..3 src has 1..3) v4 2023-10-30T12:09:21.531 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.528+0000 7feb1ffff640 3 client.4111.objecter handle_osd_map got epochs [3,3] > 0 2023-10-30T12:09:21.531 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.528+0000 7feb1ffff640 3 client.4111.objecter handle_osd_map decoding full epoch 3 2023-10-30T12:09:21.531 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.528+0000 7feb1ffff640 20 client.4111.objecter dump_active .. 0 homeless 2023-10-30T12:09:21.532 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.528+0000 7feb1ffff640 1 -- 172.21.15.92:0/728345421 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7feb2c019070 con 0x7feb380af870 2023-10-30T12:09:21.532 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.528+0000 7feb1ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:21.532 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.528+0000 7feb1ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:21.563 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:21.561+0000 7f0d0b844040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T12:09:21.600 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.599+0000 7feb3de21640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6466ea5c-d74a-4709-a83a-f5d4fe17157e", "id": 1}] 2023-10-30T12:09:21.601 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.599+0000 7feb3de21640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6466ea5c-d74a-4709-a83a-f5d4fe17157e", "id": 1}] 2023-10-30T12:09:21.601 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.599+0000 7feb3de21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:21.601 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.599+0000 7feb3de21640 1 -- 172.21.15.92:0/728345421 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6466ea5c-d74a-4709-a83a-f5d4fe17157e", "id": 1} v 0) v1 -- 0x7feb380b1f30 con 0x7feb380af870 2023-10-30T12:09:21.614 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:21.612+0000 7f67faa3e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T12:09:21.659 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:21.658+0000 7f67faa3e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T12:09:21.694 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:21.693+0000 7f0d0b844040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T12:09:21.721 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.719+0000 7feb1ffff640 1 -- 172.21.15.92:0/728345421 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6466ea5c-d74a-4709-a83a-f5d4fe17157e", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7feb2c028dd0 con 0x7feb380af870 2023-10-30T12:09:21.721 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.719+0000 7feb1ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6466ea5c-d74a-4709-a83a-f5d4fe17157e", "id": 1}] 2023-10-30T12:09:21.721 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.719+0000 7feb1ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:21.722 INFO:teuthology.orchestra.run.smithi092.stdout:1 2023-10-30T12:09:21.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.720+0000 7feb3de21640 10 librados: watch_flush enter 2023-10-30T12:09:21.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.720+0000 7feb3de21640 10 librados: watch_flush exit 2023-10-30T12:09:21.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.720+0000 7feb3de21640 20 client.4111.objecter shutdown clearing up homeless session... 2023-10-30T12:09:21.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.720+0000 7feb3de21640 10 client.4111.objecter successfully canceled tick 2023-10-30T12:09:21.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.720+0000 7feb3de21640 10 monclient: shutdown 2023-10-30T12:09:21.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.720+0000 7feb3de21640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:21.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.720+0000 7feb3de21640 1 -- 172.21.15.92:0/728345421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb380af870 msgr2=0x7feb380aca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:21.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.720+0000 7feb3de21640 1 --2- 172.21.15.92:0/728345421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb380af870 0x7feb380aca30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00d830 tx=0x7feb2c00dcf0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.725 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.720+0000 7feb3de21640 1 -- 172.21.15.92:0/728345421 shutdown_connections 2023-10-30T12:09:21.725 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.720+0000 7feb3de21640 1 --2- 172.21.15.92:0/728345421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb380a9f00 0x7feb380aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.725 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.720+0000 7feb3de21640 1 --2- 172.21.15.92:0/728345421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb380af870 0x7feb380aca30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.725 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.720+0000 7feb3de21640 1 --2- 172.21.15.92:0/728345421 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb38070ff0 0x7feb380ac4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.725 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.720+0000 7feb3de21640 1 -- 172.21.15.92:0/728345421 >> 172.21.15.92:0/728345421 conn(0x7feb380b0960 msgr2=0x7feb380b2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:21.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.721+0000 7feb3de21640 1 -- 172.21.15.92:0/728345421 shutdown_connections 2023-10-30T12:09:21.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.721+0000 7feb3de21640 1 -- 172.21.15.92:0/728345421 wait complete. 2023-10-30T12:09:21.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.721+0000 7feb3de21640 1 librados: shutdown 2023-10-30T12:09:21.732 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7 2 2023-10-30T12:09:21.744 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:21.743+0000 7f0d0b844040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T12:09:21.761 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:21.760+0000 7f67faa3e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:09:21.830 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.828+0000 7fdcfbae0640 10 monclient: get_monmap_and_config 2023-10-30T12:09:21.830 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.828+0000 7fdcfbae0640 10 monclient: build_initial_monmap 2023-10-30T12:09:21.831 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.828+0000 7fdcfbae0640 10 monclient: monmap: 2023-10-30T12:09:21.831 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:21.831 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:21.831 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:09:21.829550+0000 2023-10-30T12:09:21.831 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:09:21.829550+0000 2023-10-30T12:09:21.832 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:21.832 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:21.832 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:09:21.832 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:09:21.832 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:09:21.833 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:09:21.833 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:09:21.833 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:21.833 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.829+0000 7fdcfbae0640 1 Processor -- start 2023-10-30T12:09:21.836 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.829+0000 7fdcfbae0640 1 -- start start 2023-10-30T12:09:21.836 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.829+0000 7fdcfbae0640 10 monclient: init 2023-10-30T12:09:21.836 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.830+0000 7fdcfbae0640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:21.836 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.830+0000 7fdcfbae0640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2023-10-30T12:09:21.837 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.830+0000 7fdcfbae0640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdcf40b4440 0x7fdcf4144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:21.837 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.830+0000 7fdcfbae0640 10 monclient(hunting): picked mon.noname-a con 0x7fdcf40b4440 addr v2:172.21.15.3:3300/0 2023-10-30T12:09:21.837 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.830+0000 7fdcfbae0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdcf41453e0 addr v1:172.21.15.3:6789/0 2023-10-30T12:09:21.837 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.830+0000 7fdcfbae0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdcf40b2230 addr v1:172.21.15.92:6789/0 2023-10-30T12:09:21.837 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.830+0000 7fdcfbae0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf40a9da0 con 0x7fdcf41453e0 2023-10-30T12:09:21.838 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.830+0000 7fdcfbae0640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf40aa080 con 0x7fdcf40b2230 2023-10-30T12:09:21.838 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.830+0000 7fdcfbae0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:21.838 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.830+0000 7fdcfbae0640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcf4067a10 con 0x7fdcf40b4440 2023-10-30T12:09:21.838 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.830+0000 7fdcfbae0640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:21.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.830+0000 7fdcfbae0640 10 monclient(hunting): authenticate will time out at 1359.007348s 2023-10-30T12:09:21.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.830+0000 7fdcfa07d640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fdcf40b2230 0x7fdcf414c3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:42178/0 (socket says 172.21.15.92:42178) 2023-10-30T12:09:21.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.830+0000 7fdcfa07d640 1 -- 172.21.15.92:0/340247554 learned_addr learned my addr 172.21.15.92:0/340247554 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-10-30T12:09:21.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.830+0000 7fdcf987c640 1 --2- 172.21.15.92:0/340247554 >> v2:172.21.15.3:3300/0 conn(0x7fdcf40b4440 0x7fdcf4144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:21.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.831+0000 7fdcf987c640 10 monclient(hunting): get_auth_request con 0x7fdcf40b4440 auth_method 0 2023-10-30T12:09:21.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.831+0000 7fdcf987c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:21.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.831+0000 7fdcf987c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:21.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.831+0000 7fdcf987c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:21.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.831+0000 7fdcf987c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:21.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.831+0000 7fdcf987c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:21.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.831+0000 7fdcf987c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:21.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.833+0000 7fdcf887a640 1 -- 172.21.15.92:0/340247554 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230280563 0 0) 0x7fdcec001490 con 0x7fdcf40b2230 2023-10-30T12:09:21.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.833+0000 7fdcf887a640 10 monclient(hunting): _negotiate 2023-10-30T12:09:21.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.833+0000 7fdcf887a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:21.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.833+0000 7fdcf887a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:21.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.833+0000 7fdcf887a640 10 monclient(hunting): my global_id is 4114 2023-10-30T12:09:21.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.833+0000 7fdcf887a640 1 -- 172.21.15.92:0/340247554 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce0003520 con 0x7fdcf40b2230 2023-10-30T12:09:21.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.833+0000 7fdcf987c640 10 monclient(hunting): handle_auth_done global_id 4128 payload 274 2023-10-30T12:09:21.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.834+0000 7fdcf987c640 1 -- 172.21.15.92:0/340247554 >> v1:172.21.15.92:6789/0 conn(0x7fdcf40b2230 legacy=0x7fdcf414c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:21.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.834+0000 7fdcf987c640 1 -- 172.21.15.92:0/340247554 >> v1:172.21.15.3:6789/0 conn(0x7fdcf41453e0 legacy=0x7fdcf40b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:21.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.834+0000 7fdcf987c640 10 monclient: _finish_hunting 0 2023-10-30T12:09:21.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.834+0000 7fdcf987c640 1 monclient: found mon.noname-a 2023-10-30T12:09:21.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.834+0000 7fdcf987c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:21.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.834+0000 7fdcf987c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:09:21.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.834+0000 7fdcf987c640 1 -- 172.21.15.92:0/340247554 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf40aa360 con 0x7fdcf40b4440 2023-10-30T12:09:21.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.836+0000 7fdcf987c640 1 --2- 172.21.15.92:0/340247554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcf40b4440 0x7fdcf4144ea0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdce40097a0 tx=0x7fdce402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa905804aac67af2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:21.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcf887a640 1 -- 172.21.15.92:0/340247554 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce4031700 con 0x7fdcf40b4440 2023-10-30T12:09:21.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcf887a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:21.846 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcf887a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:21.846 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcf887a640 10 monclient: dump: 2023-10-30T12:09:21.846 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:21.846 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:21.847 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.847 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.847 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:21.847 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:21.847 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:21.848 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:21.848 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:21.848 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:21.848 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcf887a640 10 monclient: _finish_auth 0 2023-10-30T12:09:21.848 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcf887a640 10 monclient: _check_auth_tickets 2023-10-30T12:09:21.849 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcf887a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:21.849 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcf887a640 1 -- 172.21.15.92:0/340247554 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce4031870 con 0x7fdcf40b4440 2023-10-30T12:09:21.849 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcf887a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:21.849 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcf887a640 1 -- 172.21.15.92:0/340247554 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce4031c10 con 0x7fdcf40b4440 2023-10-30T12:09:21.849 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcf887a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:21.850 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcf887a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:21.850 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcf887a640 10 monclient: dump: 2023-10-30T12:09:21.850 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:21.850 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:21.850 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.851 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.851 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:21.851 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:21.851 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:21.851 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:21.851 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:21.852 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:21.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcfbae0640 5 monclient: authenticate success, global_id 4128 2023-10-30T12:09:21.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcfbae0640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:21.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.842+0000 7fdcfbae0640 4 set_mon_vals no callback set 2023-10-30T12:09:21.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.843+0000 7fdcfbae0640 10 monclient: shutdown 2023-10-30T12:09:21.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.843+0000 7fdcfbae0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:21.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.843+0000 7fdcfbae0640 1 -- 172.21.15.92:0/340247554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcf40b4440 msgr2=0x7fdcf4144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:21.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.843+0000 7fdcfbae0640 1 --2- 172.21.15.92:0/340247554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcf40b4440 0x7fdcf4144ea0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdce40097a0 tx=0x7fdce402b7a0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.843+0000 7fdcfbae0640 1 -- 172.21.15.92:0/340247554 shutdown_connections 2023-10-30T12:09:21.854 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.843+0000 7fdcfbae0640 1 --2- 172.21.15.92:0/340247554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcf40b4440 0x7fdcf4144ea0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.854 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.843+0000 7fdcfbae0640 1 -- 172.21.15.92:0/340247554 >> 172.21.15.92:0/340247554 conn(0x7fdcf4065ea0 msgr2=0x7fdcf40662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:21.854 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.843+0000 7fdcfbae0640 1 -- 172.21.15.92:0/340247554 shutdown_connections 2023-10-30T12:09:21.854 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.843+0000 7fdcfbae0640 1 -- 172.21.15.92:0/340247554 wait complete. 2023-10-30T12:09:21.854 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.843+0000 7fdcfbae0640 10 monclient: build_initial_monmap 2023-10-30T12:09:21.855 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.843+0000 7fdcfbae0640 10 monclient: monmap: 2023-10-30T12:09:21.855 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:21.855 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:21.855 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-10-30T12:09:21.855 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-10-30T12:09:21.856 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:21.856 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-10-30T12:09:21.856 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:09:21.856 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:09:21.856 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:09:21.857 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:21.857 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 1 librados: starting msgr at 2023-10-30T12:09:21.857 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 1 librados: starting objecter 2023-10-30T12:09:21.857 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 1 Processor -- start 2023-10-30T12:09:21.857 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 1 -- start start 2023-10-30T12:09:21.858 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 1 librados: setting wanted keys 2023-10-30T12:09:21.858 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 1 librados: calling monclient init 2023-10-30T12:09:21.858 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 10 monclient: init 2023-10-30T12:09:21.859 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:21.845+0000 7f0d0b844040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:09:21.862 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:21.862 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:09:21.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdcf40b2230 0x7fdcf413f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:21.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 10 monclient(hunting): picked mon.noname-b con 0x7fdcf40b2230 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:21.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcf40b4440 0x7fdcf40aadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:21.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 10 monclient(hunting): picked mon.noname-a con 0x7fdcf40b4440 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:21.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf41453e0 0x7fdcf4142380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:21.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 10 monclient(hunting): picked mon.noname-c con 0x7fdcf41453e0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:21.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:21.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf40a8ca0 con 0x7fdcf40b4440 2023-10-30T12:09:21.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:21.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf4141df0 con 0x7fdcf40b2230 2023-10-30T12:09:21.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:21.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf40acbe0 con 0x7fdcf41453e0 2023-10-30T12:09:21.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:21.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.844+0000 7fdcfbae0640 10 monclient(hunting): authenticate will time out at 1359.022135s 2023-10-30T12:09:21.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcf987c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdcf40b2230 0x7fdcf413f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:21.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcfa07d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf41453e0 0x7fdcf4142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:21.866 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcf907b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcf40b4440 0x7fdcf40aadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:21.866 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcf987c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdcf40b2230 0x7fdcf413f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:34310/0 (socket says 172.21.15.92:34310) 2023-10-30T12:09:21.866 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcf987c640 1 -- 172.21.15.92:0/2335962837 learned_addr learned my addr 172.21.15.92:0/2335962837 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-10-30T12:09:21.866 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcf987c640 10 monclient(hunting): get_auth_request con 0x7fdcf40b2230 auth_method 0 2023-10-30T12:09:21.866 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcf987c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:21.867 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcf987c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:21.867 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcf987c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:21.867 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcfa07d640 10 monclient(hunting): get_auth_request con 0x7fdcf41453e0 auth_method 0 2023-10-30T12:09:21.867 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcfa07d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:21.867 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcfa07d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:21.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcfa07d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:21.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcf907b640 10 monclient(hunting): get_auth_request con 0x7fdcf40b4440 auth_method 0 2023-10-30T12:09:21.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcf907b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:21.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcf907b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:21.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.845+0000 7fdcf907b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:21.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.846+0000 7fdcf987c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:21.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.846+0000 7fdcf987c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:21.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.846+0000 7fdcf987c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:21.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.846+0000 7fdcf907b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:21.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.846+0000 7fdcf907b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:21.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.846+0000 7fdcf907b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:21.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.846+0000 7fdcfa07d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:21.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.846+0000 7fdcfa07d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:21.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.846+0000 7fdcfa07d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:21.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.849+0000 7fdcf987c640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2023-10-30T12:09:21.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.849+0000 7fdcf987c640 1 -- 172.21.15.92:0/2335962837 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf41453e0 msgr2=0x7fdcf4142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:21.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.849+0000 7fdcf987c640 1 --2- 172.21.15.92:0/2335962837 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf41453e0 0x7fdcf4142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.849+0000 7fdcf987c640 1 -- 172.21.15.92:0/2335962837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcf40b4440 msgr2=0x7fdcf40aadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:21.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.849+0000 7fdcf987c640 1 --2- 172.21.15.92:0/2335962837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcf40b4440 0x7fdcf40aadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.849+0000 7fdcf987c640 10 monclient: _finish_hunting 0 2023-10-30T12:09:21.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.849+0000 7fdcf987c640 1 monclient: found mon.noname-b 2023-10-30T12:09:21.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.849+0000 7fdcf987c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:21.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.849+0000 7fdcf987c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:09:21.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.849+0000 7fdcf987c640 1 -- 172.21.15.92:0/2335962837 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf41428c0 con 0x7fdcf40b2230 2023-10-30T12:09:21.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.851+0000 7fdcf987c640 1 --2- 172.21.15.92:0/2335962837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdcf40b2230 0x7fdcf413f7a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdce4000c00 tx=0x7fdce40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:21.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.851+0000 7fdcea7fc640 10 client.?.objecter ms_handle_connect 0x7fdcf40b2230 2023-10-30T12:09:21.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.851+0000 7fdcea7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:21.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcea7fc640 1 -- 172.21.15.92:0/2335962837 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce4033a30 con 0x7fdcf40b2230 2023-10-30T12:09:21.874 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:21.874 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:21.874 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcea7fc640 10 monclient: dump: 2023-10-30T12:09:21.874 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:21.874 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:21.875 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.875 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.875 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:21.875 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:21.876 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:21.876 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:21.876 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:21.876 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:21.876 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcea7fc640 10 monclient: _finish_auth 0 2023-10-30T12:09:21.877 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcea7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:09:21.877 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:21.877 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcea7fc640 1 -- 172.21.15.92:0/2335962837 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce4033ba0 con 0x7fdcf40b2230 2023-10-30T12:09:21.877 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcea7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:21.877 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcea7fc640 1 -- 172.21.15.92:0/2335962837 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce403c400 con 0x7fdcf40b2230 2023-10-30T12:09:21.878 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcfbae0640 5 monclient: authenticate success, global_id 4117 2023-10-30T12:09:21.878 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcfbae0640 10 monclient: _renew_subs 2023-10-30T12:09:21.878 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcfbae0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:21.878 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcfbae0640 1 -- 172.21.15.92:0/2335962837 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf4140110 con 0x7fdcf40b2230 2023-10-30T12:09:21.879 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcebfff640 4 set_mon_vals no callback set 2023-10-30T12:09:21.879 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:21.879 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:21.879 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcea7fc640 10 monclient: dump: 2023-10-30T12:09:21.879 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:21.880 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:21.880 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.880 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:21.880 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:21.880 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:21.881 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:21.881 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:21.881 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:21.881 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:21.881 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcfbae0640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:21.881 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcfbae0640 10 monclient: _renew_subs 2023-10-30T12:09:21.882 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcfbae0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:21.882 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcfbae0640 1 -- 172.21.15.92:0/2335962837 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf4140630 con 0x7fdcf40b2230 2023-10-30T12:09:21.882 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.854+0000 7fdcfbae0640 1 librados: init done 2023-10-30T12:09:21.882 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.855+0000 7fdcfbae0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:21.883 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.855+0000 7fdcfbae0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:21.883 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.855+0000 7fdcfbae0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:21.883 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.855+0000 7fdcfbae0640 1 -- 172.21.15.92:0/2335962837 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf4140830 con 0x7fdcf40b2230 2023-10-30T12:09:21.883 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.857+0000 7fdcea7fc640 1 -- 172.21.15.92:0/2335962837 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fdce403a040 con 0x7fdcf40b2230 2023-10-30T12:09:21.883 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.857+0000 7fdcea7fc640 1 -- 172.21.15.92:0/2335962837 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fdce4051850 con 0x7fdcf40b2230 2023-10-30T12:09:21.884 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.857+0000 7fdcea7fc640 10 client.4117.objecter ms_dispatch 0x7fdcf4000b90 osd_map(4..4 src has 1..4) v4 2023-10-30T12:09:21.884 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.857+0000 7fdcea7fc640 3 client.4117.objecter handle_osd_map got epochs [4,4] > 0 2023-10-30T12:09:21.884 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.857+0000 7fdcea7fc640 3 client.4117.objecter handle_osd_map decoding full epoch 4 2023-10-30T12:09:21.884 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.857+0000 7fdcea7fc640 20 client.4117.objecter dump_active .. 0 homeless 2023-10-30T12:09:21.884 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:21.859+0000 7f67faa3e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T12:09:21.946 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:21.944+0000 7f0d0b844040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T12:09:21.954 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.953+0000 7fdcea7fc640 1 -- 172.21.15.92:0/2335962837 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fdce4051430 con 0x7fdcf40b2230 2023-10-30T12:09:21.954 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.953+0000 7fdcea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:21.955 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:21.953+0000 7fdcea7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:22.002 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:22.001+0000 7f67faa3e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:09:22.025 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.023+0000 7fdcfbae0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7", "id": 2}] 2023-10-30T12:09:22.025 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.023+0000 7fdcfbae0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7", "id": 2}] 2023-10-30T12:09:22.025 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.023+0000 7fdcfbae0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:22.025 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.023+0000 7fdcfbae0640 1 -- 172.21.15.92:0/2335962837 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7", "id": 2} v 0) v1 -- 0x7fdcf4142a60 con 0x7fdcf40b2230 2023-10-30T12:09:22.045 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:22.043+0000 7f67faa3e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T12:09:22.096 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:22.095+0000 7f0d0b844040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:09:22.106 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.105+0000 7fdcea7fc640 1 -- 172.21.15.92:0/2335962837 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdce4051a20 con 0x7fdcf40b2230 2023-10-30T12:09:22.106 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.105+0000 7fdcea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7", "id": 2}] 2023-10-30T12:09:22.106 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.105+0000 7fdcea7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:22.107 INFO:teuthology.orchestra.run.smithi092.stdout:2 2023-10-30T12:09:22.108 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.105+0000 7fdcfbae0640 10 librados: watch_flush enter 2023-10-30T12:09:22.108 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.105+0000 7fdcfbae0640 10 librados: watch_flush exit 2023-10-30T12:09:22.108 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.105+0000 7fdcfbae0640 20 client.4117.objecter shutdown clearing up homeless session... 2023-10-30T12:09:22.108 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.105+0000 7fdcfbae0640 10 client.4117.objecter successfully canceled tick 2023-10-30T12:09:22.109 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.105+0000 7fdcfbae0640 10 monclient: shutdown 2023-10-30T12:09:22.109 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.105+0000 7fdcfbae0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:22.109 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.105+0000 7fdcfbae0640 1 -- 172.21.15.92:0/2335962837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdcf40b2230 msgr2=0x7fdcf413f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:22.109 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.105+0000 7fdcfbae0640 1 --2- 172.21.15.92:0/2335962837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdcf40b2230 0x7fdcf413f7a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdce4000c00 tx=0x7fdce40046c0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.109 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.105+0000 7fdcfbae0640 1 -- 172.21.15.92:0/2335962837 shutdown_connections 2023-10-30T12:09:22.110 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.105+0000 7fdcfbae0640 1 --2- 172.21.15.92:0/2335962837 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf41453e0 0x7fdcf4142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.110 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.106+0000 7fdcfbae0640 1 --2- 172.21.15.92:0/2335962837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcf40b4440 0x7fdcf40aadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.110 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.106+0000 7fdcfbae0640 1 --2- 172.21.15.92:0/2335962837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdcf40b2230 0x7fdcf413f7a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.110 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.106+0000 7fdcfbae0640 1 -- 172.21.15.92:0/2335962837 >> 172.21.15.92:0/2335962837 conn(0x7fdcf4065ea0 msgr2=0x7fdcf413c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:22.110 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.106+0000 7fdcfbae0640 1 -- 172.21.15.92:0/2335962837 shutdown_connections 2023-10-30T12:09:22.110 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.106+0000 7fdcfbae0640 1 -- 172.21.15.92:0/2335962837 wait complete. 2023-10-30T12:09:22.111 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.106+0000 7fdcfbae0640 1 librados: shutdown 2023-10-30T12:09:22.117 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 69ec5657-9972-4f4b-8012-83886eb47429 3 2023-10-30T12:09:22.141 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:22.140+0000 7f0d0b844040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T12:09:22.191 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:22.189+0000 7f67faa3e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T12:09:22.259 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.258+0000 7fa48fdb0640 10 monclient: get_monmap_and_config 2023-10-30T12:09:22.259 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.258+0000 7fa48fdb0640 10 monclient: build_initial_monmap 2023-10-30T12:09:22.260 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.258+0000 7fa48fdb0640 10 monclient: monmap: 2023-10-30T12:09:22.260 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:22.260 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:22.260 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:09:22.259084+0000 2023-10-30T12:09:22.260 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:09:22.259084+0000 2023-10-30T12:09:22.261 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:22.261 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:22.261 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:09:22.261 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:09:22.261 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:09:22.262 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:09:22.262 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:09:22.262 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:22.264 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.258+0000 7fa48fdb0640 1 Processor -- start 2023-10-30T12:09:22.264 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 1 -- start start 2023-10-30T12:09:22.264 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 10 monclient: init 2023-10-30T12:09:22.265 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:22.265 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-10-30T12:09:22.265 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa4880b4440 0x7fa488144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:22.265 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 10 monclient(hunting): picked mon.noname-a con 0x7fa4880b4440 addr v2:172.21.15.3:3300/0 2023-10-30T12:09:22.265 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa4881453e0 0x7fa4880b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:22.266 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 10 monclient(hunting): picked mon.noname-c con 0x7fa4881453e0 addr v2:172.21.15.92:3300/0 2023-10-30T12:09:22.266 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4880b2060 addr v1:172.21.15.3:6789/0 2023-10-30T12:09:22.266 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4880a9da0 con 0x7fa4880b2060 2023-10-30T12:09:22.266 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:22.266 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa488067a10 con 0x7fa4880b4440 2023-10-30T12:09:22.267 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:22.267 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4880aa080 con 0x7fa4881453e0 2023-10-30T12:09:22.267 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:22.267 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.259+0000 7fa48fdb0640 10 monclient(hunting): authenticate will time out at 1359.436838s 2023-10-30T12:09:22.267 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.260+0000 7fa48d34b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa4881453e0 0x7fa4880b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:22.268 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.260+0000 7fa48db4c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa4880b4440 0x7fa488144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:22.268 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.260+0000 7fa48db4c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa4880b4440 0x7fa488144ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.92:36896/0 (socket says 172.21.15.92:36896) 2023-10-30T12:09:22.268 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.260+0000 7fa48d34b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa4881453e0 0x7fa4880b1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:34312/0 (socket says 172.21.15.92:34312) 2023-10-30T12:09:22.268 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.260+0000 7fa48db4c640 1 -- 172.21.15.92:0/2721287920 learned_addr learned my addr 172.21.15.92:0/2721287920 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-10-30T12:09:22.268 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.260+0000 7fa48db4c640 10 monclient(hunting): get_auth_request con 0x7fa4880b4440 auth_method 0 2023-10-30T12:09:22.268 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.260+0000 7fa48db4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:22.269 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.260+0000 7fa48db4c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:22.269 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.260+0000 7fa48db4c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:22.269 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.260+0000 7fa48d34b640 10 monclient(hunting): get_auth_request con 0x7fa4881453e0 auth_method 0 2023-10-30T12:09:22.269 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.260+0000 7fa48d34b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:22.270 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.260+0000 7fa48d34b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:22.270 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.260+0000 7fa48d34b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:22.270 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.261+0000 7fa48d34b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:22.270 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.261+0000 7fa48d34b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:22.270 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.261+0000 7fa48d34b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:22.271 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.261+0000 7fa48db4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:22.271 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.261+0000 7fa48db4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:22.271 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.261+0000 7fa48db4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:22.271 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.262+0000 7fa48d34b640 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2023-10-30T12:09:22.271 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.262+0000 7fa48d34b640 1 -- 172.21.15.92:0/2721287920 >> v2:172.21.15.3:3300/0 conn(0x7fa4880b4440 msgr2=0x7fa488144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:22.271 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.262+0000 7fa48d34b640 1 --2- 172.21.15.92:0/2721287920 >> v2:172.21.15.3:3300/0 conn(0x7fa4880b4440 0x7fa488144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.272 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.262+0000 7fa48d34b640 1 -- 172.21.15.92:0/2721287920 >> v1:172.21.15.3:6789/0 conn(0x7fa4880b2060 legacy=0x7fa48813e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:22.272 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.262+0000 7fa48d34b640 10 monclient: _finish_hunting 0 2023-10-30T12:09:22.272 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.262+0000 7fa48d34b640 1 monclient: found mon.noname-c 2023-10-30T12:09:22.272 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.262+0000 7fa48d34b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:22.272 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.262+0000 7fa48d34b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:09:22.273 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.262+0000 7fa48d34b640 1 -- 172.21.15.92:0/2721287920 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4880aa360 con 0x7fa4881453e0 2023-10-30T12:09:22.276 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.264+0000 7fa48d34b640 1 --2- 172.21.15.92:0/2721287920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4881453e0 0x7fa4880b1b20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa48806bfc0 tx=0x7fa47802b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28ce2f354dc27179 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:22.276 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48cb4a640 1 -- 172.21.15.92:0/2721287920 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47802f050 con 0x7fa4881453e0 2023-10-30T12:09:22.276 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48cb4a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:22.277 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48cb4a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:22.277 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48cb4a640 10 monclient: dump: 2023-10-30T12:09:22.277 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:22.277 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:22.277 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.278 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.278 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:22.278 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:22.278 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:22.278 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:22.279 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:22.279 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:22.279 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48cb4a640 10 monclient: _finish_auth 0 2023-10-30T12:09:22.279 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48cb4a640 10 monclient: _check_auth_tickets 2023-10-30T12:09:22.279 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48cb4a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:22.279 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48cb4a640 1 -- 172.21.15.92:0/2721287920 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa478031710 con 0x7fa4881453e0 2023-10-30T12:09:22.280 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48cb4a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:22.280 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48cb4a640 1 -- 172.21.15.92:0/2721287920 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa478031ab0 con 0x7fa4881453e0 2023-10-30T12:09:22.280 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48fdb0640 5 monclient: authenticate success, global_id 4120 2023-10-30T12:09:22.280 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48fdb0640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:22.280 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48fdb0640 4 set_mon_vals no callback set 2023-10-30T12:09:22.281 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48cb4a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:22.281 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48cb4a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:22.281 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48cb4a640 10 monclient: dump: 2023-10-30T12:09:22.281 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:22.281 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:22.282 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.282 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.282 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:22.282 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:22.282 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:22.283 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:22.283 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:22.283 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:22.283 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48fdb0640 10 monclient: shutdown 2023-10-30T12:09:22.283 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48fdb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:22.283 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48fdb0640 1 -- 172.21.15.92:0/2721287920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4881453e0 msgr2=0x7fa4880b1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:22.284 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48fdb0640 1 --2- 172.21.15.92:0/2721287920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4881453e0 0x7fa4880b1b20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa48806bfc0 tx=0x7fa47802b7b0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.284 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48fdb0640 1 -- 172.21.15.92:0/2721287920 shutdown_connections 2023-10-30T12:09:22.284 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48fdb0640 1 --2- 172.21.15.92:0/2721287920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4881453e0 0x7fa4880b1b20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.284 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48fdb0640 1 --2- 172.21.15.92:0/2721287920 >> v2:172.21.15.3:3300/0 conn(0x7fa4880b4440 0x7fa488144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.284 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48fdb0640 1 -- 172.21.15.92:0/2721287920 >> 172.21.15.92:0/2721287920 conn(0x7fa488065ea0 msgr2=0x7fa4880662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:22.285 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48fdb0640 1 -- 172.21.15.92:0/2721287920 shutdown_connections 2023-10-30T12:09:22.285 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.269+0000 7fa48fdb0640 1 -- 172.21.15.92:0/2721287920 wait complete. 2023-10-30T12:09:22.285 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.270+0000 7fa48fdb0640 10 monclient: build_initial_monmap 2023-10-30T12:09:22.285 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.270+0000 7fa48fdb0640 10 monclient: monmap: 2023-10-30T12:09:22.285 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:22.285 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:22.286 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-10-30T12:09:22.286 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-10-30T12:09:22.286 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:22.286 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-10-30T12:09:22.286 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:09:22.287 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:09:22.287 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:09:22.287 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:22.287 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.270+0000 7fa48fdb0640 1 librados: starting msgr at 2023-10-30T12:09:22.287 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.270+0000 7fa48fdb0640 1 librados: starting objecter 2023-10-30T12:09:22.288 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.270+0000 7fa48fdb0640 1 Processor -- start 2023-10-30T12:09:22.288 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 1 -- start start 2023-10-30T12:09:22.288 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 1 librados: setting wanted keys 2023-10-30T12:09:22.288 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 1 librados: calling monclient init 2023-10-30T12:09:22.288 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 10 monclient: init 2023-10-30T12:09:22.288 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:22.289 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:09:22.289 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4880b2060 0x7fa48813b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:22.289 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 10 monclient(hunting): picked mon.noname-a con 0x7fa4880b2060 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:22.289 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4880b4440 0x7fa4881da230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:22.289 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 10 monclient(hunting): picked mon.noname-b con 0x7fa4880b4440 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:22.289 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4881453e0 0x7fa4881da770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:22.290 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 10 monclient(hunting): picked mon.noname-c con 0x7fa4881453e0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:22.290 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:22.290 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4880a8b50 con 0x7fa4880b2060 2023-10-30T12:09:22.290 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:22.290 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4881403c0 con 0x7fa4880b4440 2023-10-30T12:09:22.291 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:22.291 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4881dae40 con 0x7fa4881453e0 2023-10-30T12:09:22.291 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:22.291 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.271+0000 7fa48fdb0640 10 monclient(hunting): authenticate will time out at 1359.448840s 2023-10-30T12:09:22.291 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.272+0000 7fa48d34b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4880b4440 0x7fa4881da230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:22.292 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.272+0000 7fa48db4c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4880b2060 0x7fa48813b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:22.292 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.272+0000 7fa48d34b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4880b4440 0x7fa4881da230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:34328/0 (socket says 172.21.15.92:34328) 2023-10-30T12:09:22.292 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.272+0000 7fa48d34b640 1 -- 172.21.15.92:0/2765560701 learned_addr learned my addr 172.21.15.92:0/2765560701 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-10-30T12:09:22.292 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.272+0000 7fa48d34b640 10 monclient(hunting): get_auth_request con 0x7fa4880b4440 auth_method 0 2023-10-30T12:09:22.292 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.272+0000 7fa48d34b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:22.293 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.272+0000 7fa48d34b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:22.293 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.272+0000 7fa48d34b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:22.293 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.272+0000 7fa48db4c640 10 monclient(hunting): get_auth_request con 0x7fa4880b2060 auth_method 0 2023-10-30T12:09:22.293 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.272+0000 7fa48db4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:22.293 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.272+0000 7fa48db4c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:22.293 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.272+0000 7fa48db4c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:22.294 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.273+0000 7fa48d34b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:22.294 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.273+0000 7fa48d34b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:22.294 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.273+0000 7fa48d34b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:22.294 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.273+0000 7fa48e34d640 1 --2- 172.21.15.92:0/2765560701 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4881453e0 0x7fa4881da770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:22.294 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.273+0000 7fa48db4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:22.295 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.273+0000 7fa48db4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:22.295 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.273+0000 7fa48db4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:22.295 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.274+0000 7fa48e34d640 10 monclient(hunting): get_auth_request con 0x7fa4881453e0 auth_method 0 2023-10-30T12:09:22.295 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.274+0000 7fa48e34d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:22.295 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.274+0000 7fa48e34d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:22.296 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.274+0000 7fa48e34d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:22.296 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:22.292+0000 7f0d0b844040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T12:09:22.298 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.275+0000 7fa48d34b640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2023-10-30T12:09:22.298 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.275+0000 7fa48d34b640 1 -- 172.21.15.92:0/2765560701 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4881453e0 msgr2=0x7fa4881da770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:22.298 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.275+0000 7fa48d34b640 1 --2- 172.21.15.92:0/2765560701 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4881453e0 0x7fa4881da770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.299 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.275+0000 7fa48d34b640 1 -- 172.21.15.92:0/2765560701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4880b2060 msgr2=0x7fa48813b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:22.299 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.275+0000 7fa48d34b640 1 --2- 172.21.15.92:0/2765560701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4880b2060 0x7fa48813b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.299 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.275+0000 7fa48d34b640 10 monclient: _finish_hunting 0 2023-10-30T12:09:22.299 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.275+0000 7fa48d34b640 1 monclient: found mon.noname-b 2023-10-30T12:09:22.299 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.275+0000 7fa48d34b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:22.300 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.275+0000 7fa48d34b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:09:22.300 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.275+0000 7fa48d34b640 1 -- 172.21.15.92:0/2765560701 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4881ccd50 con 0x7fa4880b4440 2023-10-30T12:09:22.300 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.277+0000 7fa48d34b640 1 --2- 172.21.15.92:0/2765560701 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4880b4440 0x7fa4881da230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa478000c00 tx=0x7fa4780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:22.300 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.277+0000 7fa47e7fc640 10 client.?.objecter ms_handle_connect 0x7fa4880b4440 2023-10-30T12:09:22.300 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.277+0000 7fa47e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:22.301 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa47e7fc640 1 -- 172.21.15.92:0/2765560701 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa478033b40 con 0x7fa4880b4440 2023-10-30T12:09:22.301 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa47e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:22.301 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa47e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:22.301 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa47e7fc640 10 monclient: dump: 2023-10-30T12:09:22.301 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:22.301 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:22.302 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.302 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.302 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:22.302 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:22.302 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:22.303 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:22.303 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:22.303 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:22.303 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa47e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:09:22.303 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa47e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:09:22.304 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa47e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:22.304 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa47e7fc640 1 -- 172.21.15.92:0/2765560701 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa478033cb0 con 0x7fa4880b4440 2023-10-30T12:09:22.304 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa48fdb0640 5 monclient: authenticate success, global_id 4123 2023-10-30T12:09:22.304 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa47e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:22.304 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa48fdb0640 10 monclient: _renew_subs 2023-10-30T12:09:22.304 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa47e7fc640 1 -- 172.21.15.92:0/2765560701 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47803c640 con 0x7fa4880b4440 2023-10-30T12:09:22.305 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa48fdb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:22.305 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa48fdb0640 1 -- 172.21.15.92:0/2765560701 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4881ccfb0 con 0x7fa4880b4440 2023-10-30T12:09:22.305 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa47ffff640 4 set_mon_vals no callback set 2023-10-30T12:09:22.305 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa47e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:22.305 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa47e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:22.306 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa47e7fc640 10 monclient: dump: 2023-10-30T12:09:22.306 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:22.306 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:22.306 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.306 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.307 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:22.307 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:22.307 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:22.307 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:22.307 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:22.307 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:22.308 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa48fdb0640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:22.308 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa48fdb0640 10 monclient: _renew_subs 2023-10-30T12:09:22.308 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa48fdb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:22.308 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa48fdb0640 1 -- 172.21.15.92:0/2765560701 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4881cd470 con 0x7fa4880b4440 2023-10-30T12:09:22.308 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.282+0000 7fa48fdb0640 1 librados: init done 2023-10-30T12:09:22.309 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.283+0000 7fa48fdb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:22.309 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.283+0000 7fa48fdb0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:22.309 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.283+0000 7fa48fdb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:22.309 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.283+0000 7fa48fdb0640 1 -- 172.21.15.92:0/2765560701 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4881ccb30 con 0x7fa4880b4440 2023-10-30T12:09:22.389 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.387+0000 7fa47e7fc640 1 -- 172.21.15.92:0/2765560701 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa47803a040 con 0x7fa4880b4440 2023-10-30T12:09:22.389 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.387+0000 7fa47e7fc640 1 -- 172.21.15.92:0/2765560701 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fa4780316b0 con 0x7fa4880b4440 2023-10-30T12:09:22.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.387+0000 7fa47e7fc640 10 client.4123.objecter ms_dispatch 0x7fa488000b90 osd_map(5..5 src has 1..5) v4 2023-10-30T12:09:22.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.387+0000 7fa47e7fc640 3 client.4123.objecter handle_osd_map got epochs [5,5] > 0 2023-10-30T12:09:22.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.387+0000 7fa47e7fc640 3 client.4123.objecter handle_osd_map decoding full epoch 5 2023-10-30T12:09:22.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.387+0000 7fa47e7fc640 20 client.4123.objecter dump_active .. 0 homeless 2023-10-30T12:09:22.391 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.387+0000 7fa47e7fc640 1 -- 172.21.15.92:0/2765560701 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fa478032960 con 0x7fa4880b4440 2023-10-30T12:09:22.391 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.387+0000 7fa47e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:22.391 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.387+0000 7fa47e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:22.464 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.463+0000 7fa48fdb0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "69ec5657-9972-4f4b-8012-83886eb47429", "id": 3}] 2023-10-30T12:09:22.465 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.463+0000 7fa48fdb0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "69ec5657-9972-4f4b-8012-83886eb47429", "id": 3}] 2023-10-30T12:09:22.465 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.463+0000 7fa48fdb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:22.465 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.463+0000 7fa48fdb0640 1 -- 172.21.15.92:0/2765560701 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "69ec5657-9972-4f4b-8012-83886eb47429", "id": 3} v 0) v1 -- 0x7fa48804e9b0 con 0x7fa4880b4440 2023-10-30T12:09:22.555 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.553+0000 7fa47e7fc640 1 -- 172.21.15.92:0/2765560701 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "69ec5657-9972-4f4b-8012-83886eb47429", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa47803cc40 con 0x7fa4880b4440 2023-10-30T12:09:22.555 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.553+0000 7fa47e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "69ec5657-9972-4f4b-8012-83886eb47429", "id": 3}] 2023-10-30T12:09:22.555 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.553+0000 7fa47e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:22.556 INFO:teuthology.orchestra.run.smithi092.stdout:3 2023-10-30T12:09:22.557 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 10 librados: watch_flush enter 2023-10-30T12:09:22.557 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 10 librados: watch_flush exit 2023-10-30T12:09:22.557 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 20 client.4123.objecter shutdown clearing up homeless session... 2023-10-30T12:09:22.558 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 10 client.4123.objecter successfully canceled tick 2023-10-30T12:09:22.558 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 10 monclient: shutdown 2023-10-30T12:09:22.558 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:22.558 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 1 -- 172.21.15.92:0/2765560701 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4880b4440 msgr2=0x7fa4881da230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:22.558 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 1 --2- 172.21.15.92:0/2765560701 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4880b4440 0x7fa4881da230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa478000c00 tx=0x7fa4780046c0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.559 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 1 -- 172.21.15.92:0/2765560701 shutdown_connections 2023-10-30T12:09:22.559 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 1 --2- 172.21.15.92:0/2765560701 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4881453e0 0x7fa4881da770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.559 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 1 --2- 172.21.15.92:0/2765560701 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4880b4440 0x7fa4881da230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.559 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 1 --2- 172.21.15.92:0/2765560701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4880b2060 0x7fa48813b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.559 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 1 -- 172.21.15.92:0/2765560701 >> 172.21.15.92:0/2765560701 conn(0x7fa488065ea0 msgr2=0x7fa48813cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:22.560 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 1 -- 172.21.15.92:0/2765560701 shutdown_connections 2023-10-30T12:09:22.560 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 1 -- 172.21.15.92:0/2765560701 wait complete. 2023-10-30T12:09:22.560 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.554+0000 7fa48fdb0640 1 librados: shutdown 2023-10-30T12:09:22.566 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new da3dad24-4aeb-47a3-ad1d-d13c77cafad0 4 2023-10-30T12:09:22.705 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.704+0000 7f7c4309b640 10 monclient: get_monmap_and_config 2023-10-30T12:09:22.706 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.704+0000 7f7c4309b640 10 monclient: build_initial_monmap 2023-10-30T12:09:22.706 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.704+0000 7f7c4309b640 10 monclient: monmap: 2023-10-30T12:09:22.706 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:22.706 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:22.707 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:09:22.704974+0000 2023-10-30T12:09:22.707 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:09:22.704974+0000 2023-10-30T12:09:22.707 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:22.707 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:22.707 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:09:22.708 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:09:22.708 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:09:22.708 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:09:22.708 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:09:22.708 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:22.710 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.704+0000 7f7c4309b640 1 Processor -- start 2023-10-30T12:09:22.711 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.704+0000 7f7c4309b640 1 -- start start 2023-10-30T12:09:22.711 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.704+0000 7f7c4309b640 10 monclient: init 2023-10-30T12:09:22.711 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.705+0000 7f7c4309b640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:22.711 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.705+0000 7f7c4309b640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-10-30T12:09:22.711 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.705+0000 7f7c4309b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c3c0b4440 addr v1:172.21.15.3:6789/0 2023-10-30T12:09:22.712 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.705+0000 7f7c4309b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7c3c1455b0 0x7f7c3c13e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:22.712 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.705+0000 7f7c4309b640 10 monclient(hunting): picked mon.noname-c con 0x7f7c3c1455b0 addr v2:172.21.15.92:3300/0 2023-10-30T12:09:22.712 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.705+0000 7f7c4309b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c3c13eb80 addr v1:172.21.15.92:6789/0 2023-10-30T12:09:22.712 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.705+0000 7f7c4309b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c3c0a9da0 con 0x7f7c3c0b4440 2023-10-30T12:09:22.712 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.705+0000 7f7c4309b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c3c0aa080 con 0x7f7c3c13eb80 2023-10-30T12:09:22.712 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.705+0000 7f7c4309b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:22.713 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.705+0000 7f7c4309b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c3c067a10 con 0x7f7c3c1455b0 2023-10-30T12:09:22.713 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.705+0000 7f7c4309b640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:22.713 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.705+0000 7f7c4309b640 10 monclient(hunting): authenticate will time out at 1359.882722s 2023-10-30T12:09:22.713 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.705+0000 7f7c3bfff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7c3c1455b0 0x7f7c3c13e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:22.713 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.706+0000 7f7c40e37640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7c3c0b4440 0x7f7c3c144ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.92:45170/0 (socket says 172.21.15.92:45170) 2023-10-30T12:09:22.714 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.706+0000 7f7c40e37640 1 -- 172.21.15.92:0/790276997 learned_addr learned my addr 172.21.15.92:0/790276997 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-10-30T12:09:22.714 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.706+0000 7f7c3bfff640 10 monclient(hunting): get_auth_request con 0x7f7c3c1455b0 auth_method 0 2023-10-30T12:09:22.714 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.706+0000 7f7c3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:22.714 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.706+0000 7f7c3bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:22.714 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.706+0000 7f7c3bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:22.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.706+0000 7f7c3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:22.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.706+0000 7f7c3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:22.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.706+0000 7f7c3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:22.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.708+0000 7f7c3bfff640 10 monclient(hunting): handle_auth_done global_id 4126 payload 274 2023-10-30T12:09:22.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.708+0000 7f7c3bfff640 1 -- 172.21.15.92:0/790276997 >> v1:172.21.15.92:6789/0 conn(0x7f7c3c13eb80 legacy=0x7f7c3c14c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:22.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.708+0000 7f7c3bfff640 1 -- 172.21.15.92:0/790276997 >> v1:172.21.15.3:6789/0 conn(0x7f7c3c0b4440 legacy=0x7f7c3c144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:22.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.708+0000 7f7c3bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:09:22.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.708+0000 7f7c3bfff640 1 monclient: found mon.noname-c 2023-10-30T12:09:22.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.708+0000 7f7c3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:22.717 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.708+0000 7f7c3bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:09:22.717 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.708+0000 7f7c3bfff640 1 -- 172.21.15.92:0/790276997 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c0aa360 con 0x7f7c3c1455b0 2023-10-30T12:09:22.719 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.710+0000 7f7c3bfff640 1 --2- 172.21.15.92:0/790276997 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c1455b0 0x7f7c3c13e640 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7c240097a0 tx=0x7f7c2402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f554863a2b228a32 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:22.719 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c3b7fe640 1 -- 172.21.15.92:0/790276997 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24031700 con 0x7f7c3c1455b0 2023-10-30T12:09:22.720 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:22.720 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c3b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:22.720 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c3b7fe640 10 monclient: dump: 2023-10-30T12:09:22.720 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:22.720 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:22.721 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.721 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.721 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:22.721 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:22.722 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:22.722 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:22.722 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:22.722 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:22.722 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c3b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:09:22.722 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c3b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:09:22.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:22.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c3b7fe640 1 -- 172.21.15.92:0/790276997 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c24031870 con 0x7f7c3c1455b0 2023-10-30T12:09:22.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c4309b640 5 monclient: authenticate success, global_id 4126 2023-10-30T12:09:22.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c4309b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:22.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c3b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:22.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c4309b640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:22.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c3b7fe640 1 -- 172.21.15.92:0/790276997 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24031c10 con 0x7f7c3c1455b0 2023-10-30T12:09:22.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:22.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c4309b640 4 set_mon_vals no callback set 2023-10-30T12:09:22.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c3b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:22.725 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.716+0000 7f7c3b7fe640 10 monclient: dump: 2023-10-30T12:09:22.725 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:22.725 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:22.725 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.725 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.725 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:22.726 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:22.726 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:22.726 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:22.726 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:22.727 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:22.727 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.717+0000 7f7c4309b640 10 monclient: shutdown 2023-10-30T12:09:22.727 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.717+0000 7f7c4309b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:22.727 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.717+0000 7f7c4309b640 1 -- 172.21.15.92:0/790276997 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c1455b0 msgr2=0x7f7c3c13e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:22.727 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.717+0000 7f7c4309b640 1 --2- 172.21.15.92:0/790276997 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c1455b0 0x7f7c3c13e640 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7c240097a0 tx=0x7f7c2402b7a0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.728 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.717+0000 7f7c4309b640 1 -- 172.21.15.92:0/790276997 shutdown_connections 2023-10-30T12:09:22.728 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.717+0000 7f7c4309b640 1 --2- 172.21.15.92:0/790276997 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c1455b0 0x7f7c3c13e640 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.728 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.717+0000 7f7c4309b640 1 -- 172.21.15.92:0/790276997 >> 172.21.15.92:0/790276997 conn(0x7f7c3c065ea0 msgr2=0x7f7c3c0662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:22.728 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.717+0000 7f7c4309b640 1 -- 172.21.15.92:0/790276997 shutdown_connections 2023-10-30T12:09:22.728 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.717+0000 7f7c4309b640 1 -- 172.21.15.92:0/790276997 wait complete. 2023-10-30T12:09:22.729 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.718+0000 7f7c4309b640 10 monclient: build_initial_monmap 2023-10-30T12:09:22.729 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.718+0000 7f7c4309b640 10 monclient: monmap: 2023-10-30T12:09:22.729 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:22.729 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:22.730 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-10-30T12:09:22.730 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-10-30T12:09:22.730 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:22.730 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-10-30T12:09:22.730 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:09:22.731 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:09:22.731 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:09:22.731 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:22.731 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.718+0000 7f7c4309b640 1 librados: starting msgr at 2023-10-30T12:09:22.731 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.718+0000 7f7c4309b640 1 librados: starting objecter 2023-10-30T12:09:22.731 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.718+0000 7f7c4309b640 1 Processor -- start 2023-10-30T12:09:22.732 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.718+0000 7f7c4309b640 1 -- start start 2023-10-30T12:09:22.732 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.718+0000 7f7c4309b640 1 librados: setting wanted keys 2023-10-30T12:09:22.732 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.718+0000 7f7c4309b640 1 librados: calling monclient init 2023-10-30T12:09:22.732 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.718+0000 7f7c4309b640 10 monclient: init 2023-10-30T12:09:22.736 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:22.737 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:09:22.737 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c0b4440 0x7f7c3c1cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:22.737 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 10 monclient(hunting): picked mon.noname-b con 0x7f7c3c0b4440 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:22.737 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c3c13eb80 0x7f7c3c1da2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:22.737 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 10 monclient(hunting): picked mon.noname-c con 0x7f7c3c13eb80 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:22.738 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c3c1455b0 0x7f7c3c13b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:22.738 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 10 monclient(hunting): picked mon.noname-a con 0x7f7c3c1455b0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:22.738 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:22.738 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c0a8cd0 con 0x7f7c3c1455b0 2023-10-30T12:09:22.738 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:22.739 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c1dacb0 con 0x7f7c3c0b4440 2023-10-30T12:09:22.739 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:22.739 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c1ce470 con 0x7f7c3c13eb80 2023-10-30T12:09:22.739 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:22.739 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c4309b640 10 monclient(hunting): authenticate will time out at 1359.896457s 2023-10-30T12:09:22.740 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c40e37640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c0b4440 0x7f7c3c1cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:22.740 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c41638640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c3c1455b0 0x7f7c3c13b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:22.740 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c3bfff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c3c13eb80 0x7f7c3c1da2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:22.740 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c40e37640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c0b4440 0x7f7c3c1cd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:34346/0 (socket says 172.21.15.92:34346) 2023-10-30T12:09:22.740 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c40e37640 1 -- 172.21.15.92:0/2938567190 learned_addr learned my addr 172.21.15.92:0/2938567190 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-10-30T12:09:22.741 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c40e37640 10 monclient(hunting): get_auth_request con 0x7f7c3c0b4440 auth_method 0 2023-10-30T12:09:22.741 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c40e37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:22.741 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c40e37640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:22.741 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.719+0000 7f7c40e37640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:22.741 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c3bfff640 10 monclient(hunting): get_auth_request con 0x7f7c3c13eb80 auth_method 0 2023-10-30T12:09:22.742 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:22.742 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c3bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:22.742 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c3bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:22.742 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c41638640 10 monclient(hunting): get_auth_request con 0x7f7c3c1455b0 auth_method 0 2023-10-30T12:09:22.742 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c41638640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:22.743 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c41638640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:22.743 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c41638640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:22.743 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c40e37640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:22.743 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c40e37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:22.743 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c40e37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:22.743 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:22.744 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:22.744 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:22.744 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c41638640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:22.744 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c41638640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:22.744 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.720+0000 7f7c41638640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:22.745 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.723+0000 7f7c40e37640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2023-10-30T12:09:22.745 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.723+0000 7f7c40e37640 1 -- 172.21.15.92:0/2938567190 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c3c13eb80 msgr2=0x7f7c3c1da2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:22.745 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.723+0000 7f7c40e37640 1 --2- 172.21.15.92:0/2938567190 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c3c13eb80 0x7f7c3c1da2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.745 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.723+0000 7f7c40e37640 1 -- 172.21.15.92:0/2938567190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c3c1455b0 msgr2=0x7f7c3c13b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:22.745 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.723+0000 7f7c40e37640 1 --2- 172.21.15.92:0/2938567190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c3c1455b0 0x7f7c3c13b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.745 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.723+0000 7f7c40e37640 10 monclient: _finish_hunting 0 2023-10-30T12:09:22.746 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.723+0000 7f7c40e37640 1 monclient: found mon.noname-b 2023-10-30T12:09:22.746 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.723+0000 7f7c40e37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:22.746 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.723+0000 7f7c40e37640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:09:22.746 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.723+0000 7f7c40e37640 1 -- 172.21.15.92:0/2938567190 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c1da820 con 0x7f7c3c0b4440 2023-10-30T12:09:22.746 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.725+0000 7f7c40e37640 1 --2- 172.21.15.92:0/2938567190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c0b4440 0x7f7c3c1cd9d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7c3400cba0 tx=0x7f7c3400a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.725+0000 7f7c397fa640 10 client.?.objecter ms_handle_connect 0x7f7c3c0b4440 2023-10-30T12:09:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.725+0000 7f7c397fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c397fa640 1 -- 172.21.15.92:0/2938567190 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3400ad40 con 0x7f7c3c0b4440 2023-10-30T12:09:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c397fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:22.748 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c397fa640 10 monclient: dump: 2023-10-30T12:09:22.748 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:22.748 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:22.748 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.748 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.749 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:22.749 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:22.749 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:22.749 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:22.749 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:22.749 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:22.750 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c397fa640 10 monclient: _finish_auth 0 2023-10-30T12:09:22.750 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c397fa640 10 monclient: _check_auth_tickets 2023-10-30T12:09:22.750 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:22.750 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c397fa640 1 -- 172.21.15.92:0/2938567190 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3400e040 con 0x7f7c3c0b4440 2023-10-30T12:09:22.750 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c4309b640 5 monclient: authenticate success, global_id 4129 2023-10-30T12:09:22.751 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c4309b640 10 monclient: _renew_subs 2023-10-30T12:09:22.751 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c4309b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:22.751 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c4309b640 1 -- 172.21.15.92:0/2938567190 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3c13b830 con 0x7f7c3c0b4440 2023-10-30T12:09:22.751 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c397fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:22.751 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c397fa640 1 -- 172.21.15.92:0/2938567190 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34012600 con 0x7f7c3c0b4440 2023-10-30T12:09:22.752 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:22.752 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:22.752 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c397fa640 10 monclient: dump: 2023-10-30T12:09:22.752 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:22.752 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:22.752 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.753 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:22.753 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:22.753 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:22.753 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:22.753 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:22.754 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:22.754 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:22.754 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c3affd640 4 set_mon_vals no callback set 2023-10-30T12:09:22.754 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c4309b640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:22.754 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c4309b640 10 monclient: _renew_subs 2023-10-30T12:09:22.755 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c4309b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:22.755 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c4309b640 1 -- 172.21.15.92:0/2938567190 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3c13bd70 con 0x7f7c3c0b4440 2023-10-30T12:09:22.755 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.728+0000 7f7c4309b640 1 librados: init done 2023-10-30T12:09:22.755 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.729+0000 7f7c4309b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:22.755 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.729+0000 7f7c4309b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:22.755 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.729+0000 7f7c4309b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:22.756 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.729+0000 7f7c4309b640 1 -- 172.21.15.92:0/2938567190 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3c1daa20 con 0x7f7c3c0b4440 2023-10-30T12:09:22.756 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.731+0000 7f7c397fa640 1 -- 172.21.15.92:0/2938567190 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7c34012a70 con 0x7f7c3c0b4440 2023-10-30T12:09:22.756 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.731+0000 7f7c397fa640 1 -- 172.21.15.92:0/2938567190 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f7c34007580 con 0x7f7c3c0b4440 2023-10-30T12:09:22.756 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.731+0000 7f7c397fa640 10 client.4129.objecter ms_dispatch 0x7f7c3c000b90 osd_map(6..6 src has 1..6) v4 2023-10-30T12:09:22.756 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.731+0000 7f7c397fa640 3 client.4129.objecter handle_osd_map got epochs [6,6] > 0 2023-10-30T12:09:22.757 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.731+0000 7f7c397fa640 3 client.4129.objecter handle_osd_map decoding full epoch 6 2023-10-30T12:09:22.757 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.731+0000 7f7c397fa640 20 client.4129.objecter dump_active .. 0 homeless 2023-10-30T12:09:22.830 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.828+0000 7f7c397fa640 1 -- 172.21.15.92:0/2938567190 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f7c34012d20 con 0x7f7c3c0b4440 2023-10-30T12:09:22.830 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.829+0000 7f7c397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:22.831 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.829+0000 7f7c397fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:22.900 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.899+0000 7f7c4309b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "da3dad24-4aeb-47a3-ad1d-d13c77cafad0", "id": 4}] 2023-10-30T12:09:22.901 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.899+0000 7f7c4309b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "da3dad24-4aeb-47a3-ad1d-d13c77cafad0", "id": 4}] 2023-10-30T12:09:22.901 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.899+0000 7f7c4309b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:22.901 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.899+0000 7f7c4309b640 1 -- 172.21.15.92:0/2938567190 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "da3dad24-4aeb-47a3-ad1d-d13c77cafad0", "id": 4} v 0) v1 -- 0x7f7c3c04e9b0 con 0x7f7c3c0b4440 2023-10-30T12:09:22.983 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.982+0000 7f7c397fa640 1 -- 172.21.15.92:0/2938567190 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "da3dad24-4aeb-47a3-ad1d-d13c77cafad0", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7c3401c440 con 0x7f7c3c0b4440 2023-10-30T12:09:22.984 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.982+0000 7f7c397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "da3dad24-4aeb-47a3-ad1d-d13c77cafad0", "id": 4}] 2023-10-30T12:09:22.984 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.982+0000 7f7c397fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:22.984 INFO:teuthology.orchestra.run.smithi092.stdout:4 2023-10-30T12:09:22.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 10 librados: watch_flush enter 2023-10-30T12:09:22.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 10 librados: watch_flush exit 2023-10-30T12:09:22.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 20 client.4129.objecter shutdown clearing up homeless session... 2023-10-30T12:09:22.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 10 client.4129.objecter successfully canceled tick 2023-10-30T12:09:22.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 10 monclient: shutdown 2023-10-30T12:09:22.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:22.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 1 -- 172.21.15.92:0/2938567190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c0b4440 msgr2=0x7f7c3c1cd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:22.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 1 --2- 172.21.15.92:0/2938567190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c0b4440 0x7f7c3c1cd9d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7c3400cba0 tx=0x7f7c3400a430 comp rx=0 tx=0).stop 2023-10-30T12:09:22.988 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 1 -- 172.21.15.92:0/2938567190 shutdown_connections 2023-10-30T12:09:22.988 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 1 --2- 172.21.15.92:0/2938567190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c3c1455b0 0x7f7c3c13b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.988 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 1 --2- 172.21.15.92:0/2938567190 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c3c13eb80 0x7f7c3c1da2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.988 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 1 --2- 172.21.15.92:0/2938567190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c0b4440 0x7f7c3c1cd9d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.988 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 1 -- 172.21.15.92:0/2938567190 >> 172.21.15.92:0/2938567190 conn(0x7f7c3c065ea0 msgr2=0x7f7c3c066280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:22.989 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 1 -- 172.21.15.92:0/2938567190 shutdown_connections 2023-10-30T12:09:22.989 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 1 -- 172.21.15.92:0/2938567190 wait complete. 2023-10-30T12:09:22.989 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:22.983+0000 7f7c4309b640 1 librados: shutdown 2023-10-30T12:09:22.995 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new d38d9d7f-b0de-48ab-a850-499c6a0a4364 5 2023-10-30T12:09:23.024 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:23.023+0000 7f67faa3e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T12:09:23.093 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.091+0000 7fd9e4ce5640 10 monclient: get_monmap_and_config 2023-10-30T12:09:23.093 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.091+0000 7fd9e4ce5640 10 monclient: build_initial_monmap 2023-10-30T12:09:23.093 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.091+0000 7fd9e4ce5640 10 monclient: monmap: 2023-10-30T12:09:23.094 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:23.094 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:23.094 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:09:23.092317+0000 2023-10-30T12:09:23.094 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:09:23.092317+0000 2023-10-30T12:09:23.094 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:23.095 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:23.095 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:09:23.095 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:09:23.095 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:09:23.095 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:09:23.096 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:09:23.096 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:23.096 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.092+0000 7fd9e4ce5640 1 Processor -- start 2023-10-30T12:09:23.096 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.092+0000 7fd9e4ce5640 1 -- start start 2023-10-30T12:09:23.096 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.092+0000 7fd9e4ce5640 10 monclient: init 2023-10-30T12:09:23.099 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.092+0000 7fd9e4ce5640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:23.099 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.092+0000 7fd9e4ce5640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-10-30T12:09:23.099 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.092+0000 7fd9e4ce5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd9e00b4440 addr v1:172.21.15.92:6789/0 2023-10-30T12:09:23.100 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.092+0000 7fd9e4ce5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd9e01455b0 addr v1:172.21.15.3:6789/0 2023-10-30T12:09:23.100 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.092+0000 7fd9e4ce5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e013ed50 0x7fd9e014d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:23.100 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.092+0000 7fd9e4ce5640 10 monclient(hunting): picked mon.noname-b con 0x7fd9e013ed50 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:23.101 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.092+0000 7fd9e4ce5640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e00a9da0 con 0x7fd9e01455b0 2023-10-30T12:09:23.101 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.092+0000 7fd9e4ce5640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e00aa080 con 0x7fd9e00b4440 2023-10-30T12:09:23.101 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.093+0000 7fd9e4ce5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:23.101 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.093+0000 7fd9e4ce5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e0067a10 con 0x7fd9e013ed50 2023-10-30T12:09:23.101 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.093+0000 7fd9e4ce5640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:23.102 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.093+0000 7fd9e4ce5640 10 monclient(hunting): authenticate will time out at 1360.270276s 2023-10-30T12:09:23.102 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.093+0000 7fd9de59c640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fd9e00b4440 0x7fd9e0144ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:42188/0 (socket says 172.21.15.92:42188) 2023-10-30T12:09:23.102 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.093+0000 7fd9de59c640 1 -- 172.21.15.92:0/285407678 learned_addr learned my addr 172.21.15.92:0/285407678 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-10-30T12:09:23.102 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.093+0000 7fd9ded9d640 1 --2- 172.21.15.92:0/285407678 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e013ed50 0x7fd9e014d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:23.103 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.094+0000 7fd9ded9d640 10 monclient(hunting): get_auth_request con 0x7fd9e013ed50 auth_method 0 2023-10-30T12:09:23.103 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.094+0000 7fd9ded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:23.103 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.094+0000 7fd9ded9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:23.103 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.094+0000 7fd9ded9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:23.104 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.094+0000 7fd9ded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:23.104 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.094+0000 7fd9ded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:23.104 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.094+0000 7fd9ded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:23.104 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.095+0000 7fd9dd59a640 1 -- 172.21.15.92:0/285407678 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215200130 0 0) 0x7fd9d0001110 con 0x7fd9e00b4440 2023-10-30T12:09:23.105 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.095+0000 7fd9dd59a640 10 monclient(hunting): _negotiate 2023-10-30T12:09:23.105 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.095+0000 7fd9dd59a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:23.105 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.095+0000 7fd9dd59a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:23.105 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.095+0000 7fd9dd59a640 10 monclient(hunting): my global_id is 4132 2023-10-30T12:09:23.105 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.095+0000 7fd9dd59a640 1 -- 172.21.15.92:0/285407678 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c8003520 con 0x7fd9e00b4440 2023-10-30T12:09:23.106 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.096+0000 7fd9ded9d640 10 monclient(hunting): handle_auth_done global_id 4121 payload 274 2023-10-30T12:09:23.106 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.096+0000 7fd9ded9d640 1 -- 172.21.15.92:0/285407678 >> v1:172.21.15.92:6789/0 conn(0x7fd9e00b4440 legacy=0x7fd9e0144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:23.106 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.096+0000 7fd9ded9d640 1 -- 172.21.15.92:0/285407678 >> v1:172.21.15.3:6789/0 conn(0x7fd9e01455b0 legacy=0x7fd9e013e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:23.106 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.096+0000 7fd9ded9d640 10 monclient: _finish_hunting 0 2023-10-30T12:09:23.106 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.096+0000 7fd9ded9d640 1 monclient: found mon.noname-b 2023-10-30T12:09:23.107 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.096+0000 7fd9ded9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:23.107 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.096+0000 7fd9ded9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:09:23.107 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.096+0000 7fd9ded9d640 1 -- 172.21.15.92:0/285407678 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e00aa360 con 0x7fd9e013ed50 2023-10-30T12:09:23.113 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.098+0000 7fd9ded9d640 1 --2- 172.21.15.92:0/285407678 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e013ed50 0x7fd9e014d3f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009dc0 tx=0x7fd9d802be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f0d2a9096989d8c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:23.114 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9dd59a640 1 -- 172.21.15.92:0/285407678 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d802cbb0 con 0x7fd9e013ed50 2023-10-30T12:09:23.114 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:23.115 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9dd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:23.115 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9dd59a640 10 monclient: dump: 2023-10-30T12:09:23.115 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:23.116 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:23.116 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.116 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.117 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:23.117 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:23.118 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:23.118 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:23.118 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:23.118 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:23.119 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9dd59a640 10 monclient: _finish_auth 0 2023-10-30T12:09:23.119 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9dd59a640 10 monclient: _check_auth_tickets 2023-10-30T12:09:23.119 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:23.120 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9dd59a640 1 -- 172.21.15.92:0/285407678 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d802cd20 con 0x7fd9e013ed50 2023-10-30T12:09:23.120 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9e4ce5640 5 monclient: authenticate success, global_id 4121 2023-10-30T12:09:23.120 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9e4ce5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:23.120 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9dd59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:23.121 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9e4ce5640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:23.121 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9e4ce5640 4 set_mon_vals no callback set 2023-10-30T12:09:23.122 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9dd59a640 1 -- 172.21.15.92:0/285407678 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8034660 con 0x7fd9e013ed50 2023-10-30T12:09:23.122 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:23.122 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9dd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:23.122 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.102+0000 7fd9dd59a640 10 monclient: dump: 2023-10-30T12:09:23.123 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:23.123 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:23.123 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.123 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.124 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:23.124 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:23.124 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:23.125 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:23.125 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:23.125 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:23.126 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.103+0000 7fd9e4ce5640 10 monclient: shutdown 2023-10-30T12:09:23.126 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.103+0000 7fd9e4ce5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:23.126 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.103+0000 7fd9e4ce5640 1 -- 172.21.15.92:0/285407678 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e013ed50 msgr2=0x7fd9e014d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:23.127 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.103+0000 7fd9e4ce5640 1 --2- 172.21.15.92:0/285407678 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e013ed50 0x7fd9e014d3f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009dc0 tx=0x7fd9d802be00 comp rx=0 tx=0).stop 2023-10-30T12:09:23.127 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.103+0000 7fd9e4ce5640 1 -- 172.21.15.92:0/285407678 shutdown_connections 2023-10-30T12:09:23.128 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.103+0000 7fd9e4ce5640 1 --2- 172.21.15.92:0/285407678 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e013ed50 0x7fd9e014d3f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:23.128 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.103+0000 7fd9e4ce5640 1 -- 172.21.15.92:0/285407678 >> 172.21.15.92:0/285407678 conn(0x7fd9e0065ea0 msgr2=0x7fd9e00662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:23.128 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.103+0000 7fd9e4ce5640 1 -- 172.21.15.92:0/285407678 shutdown_connections 2023-10-30T12:09:23.128 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.103+0000 7fd9e4ce5640 1 -- 172.21.15.92:0/285407678 wait complete. 2023-10-30T12:09:23.129 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 10 monclient: build_initial_monmap 2023-10-30T12:09:23.129 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 10 monclient: monmap: 2023-10-30T12:09:23.129 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:23.129 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:23.130 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-10-30T12:09:23.130 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-10-30T12:09:23.130 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:23.131 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-10-30T12:09:23.131 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:09:23.131 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:09:23.133 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:09:23.133 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:23.133 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 1 librados: starting msgr at 2023-10-30T12:09:23.133 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 1 librados: starting objecter 2023-10-30T12:09:23.134 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 1 Processor -- start 2023-10-30T12:09:23.134 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 1 -- start start 2023-10-30T12:09:23.134 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 1 librados: setting wanted keys 2023-10-30T12:09:23.134 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 1 librados: calling monclient init 2023-10-30T12:09:23.135 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 10 monclient: init 2023-10-30T12:09:23.135 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:23.135 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:09:23.135 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e00b4440 0x7fd9e01435f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:23.136 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 10 monclient(hunting): picked mon.noname-c con 0x7fd9e00b4440 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:23.136 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9e013ed50 0x7fd9e00b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:23.136 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 10 monclient(hunting): picked mon.noname-a con 0x7fd9e013ed50 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:23.137 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9e01455b0 0x7fd9e01d6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:23.137 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 10 monclient(hunting): picked mon.noname-b con 0x7fd9e01455b0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:23.137 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:23.137 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e00a8e20 con 0x7fd9e013ed50 2023-10-30T12:09:23.137 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:23.138 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e01da450 con 0x7fd9e01455b0 2023-10-30T12:09:23.138 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:23.138 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e01d70b0 con 0x7fd9e00b4440 2023-10-30T12:09:23.138 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:23.138 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.104+0000 7fd9e4ce5640 10 monclient(hunting): authenticate will time out at 1360.282197s 2023-10-30T12:09:23.139 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.105+0000 7fd9ded9d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9e01455b0 0x7fd9e01d6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:23.139 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.105+0000 7fd9de59c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e00b4440 0x7fd9e01435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:23.139 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.105+0000 7fd9ddd9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9e013ed50 0x7fd9e00b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:23.139 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.105+0000 7fd9ded9d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9e01455b0 0x7fd9e01d6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:34350/0 (socket says 172.21.15.92:34350) 2023-10-30T12:09:23.139 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.105+0000 7fd9ded9d640 1 -- 172.21.15.92:0/1437218697 learned_addr learned my addr 172.21.15.92:0/1437218697 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-10-30T12:09:23.140 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.105+0000 7fd9ded9d640 10 monclient(hunting): get_auth_request con 0x7fd9e01455b0 auth_method 0 2023-10-30T12:09:23.140 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.105+0000 7fd9ded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:23.140 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.105+0000 7fd9ded9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:23.140 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.105+0000 7fd9ded9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:23.140 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.105+0000 7fd9de59c640 10 monclient(hunting): get_auth_request con 0x7fd9e00b4440 auth_method 0 2023-10-30T12:09:23.140 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.105+0000 7fd9de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:23.141 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.105+0000 7fd9de59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:23.141 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.105+0000 7fd9de59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:23.141 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.106+0000 7fd9ddd9b640 10 monclient(hunting): get_auth_request con 0x7fd9e013ed50 auth_method 0 2023-10-30T12:09:23.142 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.106+0000 7fd9ddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:23.142 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.106+0000 7fd9ddd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:23.142 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.106+0000 7fd9ddd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:23.143 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.106+0000 7fd9ded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:23.143 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.106+0000 7fd9ded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:23.144 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.106+0000 7fd9ded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:23.144 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.106+0000 7fd9de59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:23.144 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.106+0000 7fd9de59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:23.145 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.106+0000 7fd9de59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:23.145 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.106+0000 7fd9ddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:23.145 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.106+0000 7fd9ddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:23.146 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.106+0000 7fd9ddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:23.146 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.109+0000 7fd9de59c640 10 monclient(hunting): handle_auth_done global_id 4124 payload 824 2023-10-30T12:09:23.147 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.109+0000 7fd9de59c640 1 -- 172.21.15.92:0/1437218697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9e01455b0 msgr2=0x7fd9e01d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:23.147 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.109+0000 7fd9de59c640 1 --2- 172.21.15.92:0/1437218697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9e01455b0 0x7fd9e01d6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:23.147 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.109+0000 7fd9de59c640 1 -- 172.21.15.92:0/1437218697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9e013ed50 msgr2=0x7fd9e00b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:23.148 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.109+0000 7fd9de59c640 1 --2- 172.21.15.92:0/1437218697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9e013ed50 0x7fd9e00b0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:23.148 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.109+0000 7fd9de59c640 10 monclient: _finish_hunting 0 2023-10-30T12:09:23.148 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.109+0000 7fd9de59c640 1 monclient: found mon.noname-c 2023-10-30T12:09:23.149 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.109+0000 7fd9de59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:23.149 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.109+0000 7fd9de59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.3:3301/0 2023-10-30T12:09:23.150 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.109+0000 7fd9de59c640 1 -- 172.21.15.92:0/1437218697 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e01cbce0 con 0x7fd9e00b4440 2023-10-30T12:09:23.150 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.111+0000 7fd9de59c640 1 --2- 172.21.15.92:0/1437218697 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e00b4440 0x7fd9e01435f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd9d000c730 tx=0x7fd9d000cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:23.150 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.111+0000 7fd9ceffd640 10 client.?.objecter ms_handle_connect 0x7fd9e00b4440 2023-10-30T12:09:23.151 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.111+0000 7fd9ceffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:23.152 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:23.137+0000 7f0d0b844040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T12:09:23.156 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9ceffd640 1 -- 172.21.15.92:0/1437218697 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d00058b0 con 0x7fd9e00b4440 2023-10-30T12:09:23.156 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:23.156 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9ceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:23.156 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9ceffd640 10 monclient: dump: 2023-10-30T12:09:23.157 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:23.157 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:23.157 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.157 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.157 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:23.158 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:23.158 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:23.158 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:23.158 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:23.158 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:23.159 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9ceffd640 10 monclient: _finish_auth 0 2023-10-30T12:09:23.159 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9ceffd640 10 monclient: _check_auth_tickets 2023-10-30T12:09:23.159 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:23.159 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9ceffd640 1 -- 172.21.15.92:0/1437218697 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d000e040 con 0x7fd9e00b4440 2023-10-30T12:09:23.159 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9ceffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:23.160 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9ceffd640 1 -- 172.21.15.92:0/1437218697 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0005ca0 con 0x7fd9e00b4440 2023-10-30T12:09:23.160 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:23.160 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:23.160 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9ceffd640 10 monclient: dump: 2023-10-30T12:09:23.161 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:23.161 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:23.161 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.161 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.161 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:23.162 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:23.162 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:23.162 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:23.162 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:23.162 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:23.163 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9e4ce5640 5 monclient: authenticate success, global_id 4124 2023-10-30T12:09:23.163 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9e4ce5640 10 monclient: _renew_subs 2023-10-30T12:09:23.163 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9e4ce5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:09:23.163 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9e4ce5640 1 -- 172.21.15.92:0/1437218697 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e01da070 con 0x7fd9e00b4440 2023-10-30T12:09:23.163 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9dcd99640 4 set_mon_vals no callback set 2023-10-30T12:09:23.164 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9e4ce5640 10 client.4124.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:23.164 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9e4ce5640 10 monclient: _renew_subs 2023-10-30T12:09:23.164 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9e4ce5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:09:23.164 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9e4ce5640 1 -- 172.21.15.92:0/1437218697 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e01dab50 con 0x7fd9e00b4440 2023-10-30T12:09:23.164 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.115+0000 7fd9e4ce5640 1 librados: init done 2023-10-30T12:09:23.164 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.116+0000 7fd9e4ce5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:23.165 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.116+0000 7fd9e4ce5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:23.165 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.116+0000 7fd9e4ce5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:09:23.165 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.116+0000 7fd9e4ce5640 1 -- 172.21.15.92:0/1437218697 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e013da60 con 0x7fd9e00b4440 2023-10-30T12:09:23.165 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.119+0000 7fd9ceffd640 1 -- 172.21.15.92:0/1437218697 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd9d0007370 con 0x7fd9e00b4440 2023-10-30T12:09:23.166 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.119+0000 7fd9ceffd640 1 -- 172.21.15.92:0/1437218697 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fd9d000ab80 con 0x7fd9e00b4440 2023-10-30T12:09:23.166 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.119+0000 7fd9ceffd640 10 client.4124.objecter ms_dispatch 0x7fd9e0000b90 osd_map(7..7 src has 1..7) v4 2023-10-30T12:09:23.166 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.119+0000 7fd9ceffd640 3 client.4124.objecter handle_osd_map got epochs [7,7] > 0 2023-10-30T12:09:23.166 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.119+0000 7fd9ceffd640 3 client.4124.objecter handle_osd_map decoding full epoch 7 2023-10-30T12:09:23.166 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.119+0000 7fd9ceffd640 20 client.4124.objecter dump_active .. 0 homeless 2023-10-30T12:09:23.241 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.240+0000 7fd9ceffd640 1 -- 172.21.15.92:0/1437218697 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fd9d000ada0 con 0x7fd9e00b4440 2023-10-30T12:09:23.241 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.240+0000 7fd9ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:23.242 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.240+0000 7fd9ceffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:23.300 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:23.299+0000 7f67faa3e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T12:09:23.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.312+0000 7fd9e4ce5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d38d9d7f-b0de-48ab-a850-499c6a0a4364", "id": 5}] 2023-10-30T12:09:23.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.312+0000 7fd9e4ce5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d38d9d7f-b0de-48ab-a850-499c6a0a4364", "id": 5}] 2023-10-30T12:09:23.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.312+0000 7fd9e4ce5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:09:23.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.312+0000 7fd9e4ce5640 1 -- 172.21.15.92:0/1437218697 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d38d9d7f-b0de-48ab-a850-499c6a0a4364", "id": 5} v 0) v1 -- 0x7fd9e01da270 con 0x7fd9e00b4440 2023-10-30T12:09:23.352 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:23.350+0000 7f67faa3e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T12:09:23.371 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.370+0000 7fd9ceffd640 1 -- 172.21.15.92:0/1437218697 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d38d9d7f-b0de-48ab-a850-499c6a0a4364", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd9d000a5e0 con 0x7fd9e00b4440 2023-10-30T12:09:23.371 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.370+0000 7fd9ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d38d9d7f-b0de-48ab-a850-499c6a0a4364", "id": 5}] 2023-10-30T12:09:23.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.370+0000 7fd9ceffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:23.372 INFO:teuthology.orchestra.run.smithi092.stdout:5 2023-10-30T12:09:23.373 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.370+0000 7fd9e4ce5640 10 librados: watch_flush enter 2023-10-30T12:09:23.373 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.370+0000 7fd9e4ce5640 10 librados: watch_flush exit 2023-10-30T12:09:23.374 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.370+0000 7fd9e4ce5640 20 client.4124.objecter shutdown clearing up homeless session... 2023-10-30T12:09:23.374 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.370+0000 7fd9e4ce5640 10 client.4124.objecter successfully canceled tick 2023-10-30T12:09:23.374 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.370+0000 7fd9e4ce5640 10 monclient: shutdown 2023-10-30T12:09:23.374 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.370+0000 7fd9e4ce5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:23.375 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.370+0000 7fd9e4ce5640 1 -- 172.21.15.92:0/1437218697 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e00b4440 msgr2=0x7fd9e01435f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:23.375 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.370+0000 7fd9e4ce5640 1 --2- 172.21.15.92:0/1437218697 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e00b4440 0x7fd9e01435f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd9d000c730 tx=0x7fd9d000cbf0 comp rx=0 tx=0).stop 2023-10-30T12:09:23.375 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.371+0000 7fd9e4ce5640 1 -- 172.21.15.92:0/1437218697 shutdown_connections 2023-10-30T12:09:23.375 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.371+0000 7fd9e4ce5640 1 --2- 172.21.15.92:0/1437218697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9e01455b0 0x7fd9e01d6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:23.375 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.371+0000 7fd9e4ce5640 1 --2- 172.21.15.92:0/1437218697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9e013ed50 0x7fd9e00b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:23.376 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.371+0000 7fd9e4ce5640 1 --2- 172.21.15.92:0/1437218697 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e00b4440 0x7fd9e01435f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:23.376 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.371+0000 7fd9e4ce5640 1 -- 172.21.15.92:0/1437218697 >> 172.21.15.92:0/1437218697 conn(0x7fd9e0065ea0 msgr2=0x7fd9e013cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:23.376 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.371+0000 7fd9e4ce5640 1 -- 172.21.15.92:0/1437218697 shutdown_connections 2023-10-30T12:09:23.376 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.371+0000 7fd9e4ce5640 1 -- 172.21.15.92:0/1437218697 wait complete. 2023-10-30T12:09:23.377 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.371+0000 7fd9e4ce5640 1 librados: shutdown 2023-10-30T12:09:23.382 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new b39d2b0a-84ef-4909-bac7-47f146305a79 6 2023-10-30T12:09:23.431 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:23.429+0000 7f0d0b844040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T12:09:23.457 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:23.456+0000 7f67faa3e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T12:09:23.486 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:23.485+0000 7f0d0b844040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T12:09:23.522 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.520+0000 7f995b1a9640 10 monclient: get_monmap_and_config 2023-10-30T12:09:23.522 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.520+0000 7f995b1a9640 10 monclient: build_initial_monmap 2023-10-30T12:09:23.522 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.520+0000 7f995b1a9640 10 monclient: monmap: 2023-10-30T12:09:23.523 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:23.523 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:23.523 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:09:23.521588+0000 2023-10-30T12:09:23.523 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:09:23.521588+0000 2023-10-30T12:09:23.523 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:23.524 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:23.524 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:09:23.524 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:09:23.524 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:09:23.525 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:09:23.525 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:09:23.525 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:23.528 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.521+0000 7f995b1a9640 1 Processor -- start 2023-10-30T12:09:23.528 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.521+0000 7f995b1a9640 1 -- start start 2023-10-30T12:09:23.528 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.521+0000 7f995b1a9640 10 monclient: init 2023-10-30T12:09:23.529 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.522+0000 7f995b1a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:23.529 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.522+0000 7f995b1a9640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-10-30T12:09:23.529 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.522+0000 7f995b1a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9954071760 addr v1:172.21.15.3:6789/0 2023-10-30T12:09:23.529 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.522+0000 7f995b1a9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9954071b40 0x7f995413bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:23.529 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.522+0000 7f995b1a9640 10 monclient(hunting): picked mon.noname-b con 0x7f9954071b40 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:23.530 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.522+0000 7f995b1a9640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f995413e610 0x7f995413eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:23.530 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.522+0000 7f995b1a9640 10 monclient(hunting): picked mon.noname-c con 0x7f995413e610 addr v2:172.21.15.92:3300/0 2023-10-30T12:09:23.530 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.522+0000 7f995b1a9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9954070fd0 con 0x7f9954071760 2023-10-30T12:09:23.530 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.522+0000 7f995b1a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:23.531 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.522+0000 7f995b1a9640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9954067b90 con 0x7f995413e610 2023-10-30T12:09:23.531 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.522+0000 7f995b1a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:23.531 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.522+0000 7f995b1a9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99540ae720 con 0x7f9954071b40 2023-10-30T12:09:23.531 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.522+0000 7f995b1a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:23.532 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.522+0000 7f995b1a9640 10 monclient(hunting): authenticate will time out at 1360.699560s 2023-10-30T12:09:23.532 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.522+0000 7f9959746640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f995413e610 0x7f995413eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:23.532 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.523+0000 7f9958f45640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9954071760 0x7f99540acc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.92:45192/0 (socket says 172.21.15.92:45192) 2023-10-30T12:09:23.532 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.523+0000 7f9958f45640 1 -- 172.21.15.92:0/1108199209 learned_addr learned my addr 172.21.15.92:0/1108199209 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-10-30T12:09:23.532 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.523+0000 7f9959746640 10 monclient(hunting): get_auth_request con 0x7f995413e610 auth_method 0 2023-10-30T12:09:23.533 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.523+0000 7f9959746640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:23.533 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.523+0000 7f9959746640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:23.533 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.523+0000 7f9959746640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:23.533 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.523+0000 7f9959746640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:23.533 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.523+0000 7f9959746640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:23.534 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.523+0000 7f9959746640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:23.534 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.525+0000 7f9959746640 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2023-10-30T12:09:23.534 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.525+0000 7f9959746640 1 -- 172.21.15.92:0/1108199209 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9954071b40 msgr2=0x7f995413bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:23.534 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.525+0000 7f9959746640 1 --2- 172.21.15.92:0/1108199209 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9954071b40 0x7f995413bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:23.535 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.525+0000 7f9959746640 1 -- 172.21.15.92:0/1108199209 >> v1:172.21.15.3:6789/0 conn(0x7f9954071760 legacy=0x7f99540acc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:23.535 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.525+0000 7f9959746640 10 monclient: _finish_hunting 0 2023-10-30T12:09:23.535 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.525+0000 7f9959746640 1 monclient: found mon.noname-c 2023-10-30T12:09:23.535 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.525+0000 7f9959746640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:23.536 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.525+0000 7f9959746640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:09:23.536 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.525+0000 7f9959746640 1 -- 172.21.15.92:0/1108199209 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9954071240 con 0x7f995413e610 2023-10-30T12:09:23.536 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.527+0000 7f9959746640 1 --2- 172.21.15.92:0/1108199209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f995413e610 0x7f995413eac0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f994c009d90 tx=0x7f994c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4383614d3ec5d31b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:23.541 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f99537fe640 1 -- 172.21.15.92:0/1108199209 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994c02cbc0 con 0x7f995413e610 2023-10-30T12:09:23.541 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f99537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:23.541 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f99537fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:23.541 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f99537fe640 10 monclient: dump: 2023-10-30T12:09:23.542 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:23.542 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:23.542 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.542 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.543 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:23.543 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:23.543 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:23.543 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:23.544 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:23.544 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:23.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f99537fe640 10 monclient: _finish_auth 0 2023-10-30T12:09:23.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f99537fe640 10 monclient: _check_auth_tickets 2023-10-30T12:09:23.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f99537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:23.545 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f995b1a9640 5 monclient: authenticate success, global_id 4138 2023-10-30T12:09:23.545 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f995b1a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:23.545 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f99537fe640 1 -- 172.21.15.92:0/1108199209 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994c02cd30 con 0x7f995413e610 2023-10-30T12:09:23.545 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f99537fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:23.545 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f995b1a9640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:23.546 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f99537fe640 1 -- 172.21.15.92:0/1108199209 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994c034660 con 0x7f995413e610 2023-10-30T12:09:23.546 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f995b1a9640 4 set_mon_vals no callback set 2023-10-30T12:09:23.546 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f99537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:23.546 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f99537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:23.547 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.530+0000 7f99537fe640 10 monclient: dump: 2023-10-30T12:09:23.547 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:23.547 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:23.547 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.547 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.548 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:23.548 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:23.548 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:23.548 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:23.548 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:23.549 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:23.549 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.531+0000 7f995b1a9640 10 monclient: shutdown 2023-10-30T12:09:23.549 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.531+0000 7f995b1a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:23.549 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.531+0000 7f995b1a9640 1 -- 172.21.15.92:0/1108199209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f995413e610 msgr2=0x7f995413eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:23.549 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.531+0000 7f995b1a9640 1 --2- 172.21.15.92:0/1108199209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f995413e610 0x7f995413eac0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f994c009d90 tx=0x7f994c02be10 comp rx=0 tx=0).stop 2023-10-30T12:09:23.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.531+0000 7f995b1a9640 1 -- 172.21.15.92:0/1108199209 shutdown_connections 2023-10-30T12:09:23.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.531+0000 7f995b1a9640 1 --2- 172.21.15.92:0/1108199209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f995413e610 0x7f995413eac0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:23.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.531+0000 7f995b1a9640 1 --2- 172.21.15.92:0/1108199209 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9954071b40 0x7f995413bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:23.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.531+0000 7f995b1a9640 1 -- 172.21.15.92:0/1108199209 >> 172.21.15.92:0/1108199209 conn(0x7f9954069d50 msgr2=0x7f995406a130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:23.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.531+0000 7f995b1a9640 1 -- 172.21.15.92:0/1108199209 shutdown_connections 2023-10-30T12:09:23.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.531+0000 7f995b1a9640 1 -- 172.21.15.92:0/1108199209 wait complete. 2023-10-30T12:09:23.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.531+0000 7f995b1a9640 10 monclient: build_initial_monmap 2023-10-30T12:09:23.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.531+0000 7f995b1a9640 10 monclient: monmap: 2023-10-30T12:09:23.552 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:23.552 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:23.552 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-10-30T12:09:23.552 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-10-30T12:09:23.552 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:23.553 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-10-30T12:09:23.553 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:09:23.553 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:09:23.553 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:09:23.553 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:23.554 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 1 librados: starting msgr at 2023-10-30T12:09:23.554 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 1 librados: starting objecter 2023-10-30T12:09:23.554 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 1 Processor -- start 2023-10-30T12:09:23.554 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 1 -- start start 2023-10-30T12:09:23.555 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 1 librados: setting wanted keys 2023-10-30T12:09:23.555 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 1 librados: calling monclient init 2023-10-30T12:09:23.555 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 10 monclient: init 2023-10-30T12:09:23.555 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:23.555 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:09:23.556 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9954071760 0x7f99541d75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:23.556 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 10 monclient(hunting): picked mon.noname-a con 0x7f9954071760 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:23.556 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f995413e610 0x7f995413a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:23.556 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 10 monclient(hunting): picked mon.noname-b con 0x7f995413e610 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:23.556 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f99541327d0 0x7f995413aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:23.557 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 10 monclient(hunting): picked mon.noname-c con 0x7f99541327d0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:23.557 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:23.557 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995406a880 con 0x7f9954071760 2023-10-30T12:09:23.557 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:23.558 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99540a8620 con 0x7f995413e610 2023-10-30T12:09:23.558 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:23.558 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99541d70c0 con 0x7f99541327d0 2023-10-30T12:09:23.558 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:23.558 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.532+0000 7f995b1a9640 10 monclient(hunting): authenticate will time out at 1360.710168s 2023-10-30T12:09:23.559 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.533+0000 7f9953fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f995413e610 0x7f995413a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:23.559 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.533+0000 7f9958f45640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9954071760 0x7f99541d75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:23.559 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.533+0000 7f9953fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f995413e610 0x7f995413a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:34356/0 (socket says 172.21.15.92:34356) 2023-10-30T12:09:23.559 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.533+0000 7f9953fff640 1 -- 172.21.15.92:0/179290651 learned_addr learned my addr 172.21.15.92:0/179290651 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-10-30T12:09:23.560 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.533+0000 7f9953fff640 10 monclient(hunting): get_auth_request con 0x7f995413e610 auth_method 0 2023-10-30T12:09:23.560 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.533+0000 7f9953fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:23.560 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.533+0000 7f9953fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:23.560 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.533+0000 7f9953fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:23.560 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.533+0000 7f9958f45640 10 monclient(hunting): get_auth_request con 0x7f9954071760 auth_method 0 2023-10-30T12:09:23.561 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.534+0000 7f9958f45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:23.561 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.534+0000 7f9958f45640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:23.561 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.534+0000 7f9958f45640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:23.561 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.534+0000 7f9953fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:23.561 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.534+0000 7f9953fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:23.562 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.534+0000 7f9953fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:23.562 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.534+0000 7f9958f45640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:23.562 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.534+0000 7f9958f45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:23.562 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.534+0000 7f9958f45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:23.563 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.536+0000 7f9953fff640 10 monclient(hunting): handle_auth_done global_id 4141 payload 824 2023-10-30T12:09:23.563 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.536+0000 7f9953fff640 1 -- 172.21.15.92:0/179290651 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f99541327d0 msgr2=0x7f995413aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:23.563 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.536+0000 7f9953fff640 1 --2- 172.21.15.92:0/179290651 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f99541327d0 0x7f995413aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:23.563 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.536+0000 7f9953fff640 1 -- 172.21.15.92:0/179290651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9954071760 msgr2=0x7f99541d75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:23.563 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.536+0000 7f9953fff640 1 --2- 172.21.15.92:0/179290651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9954071760 0x7f99541d75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:23.564 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.536+0000 7f9953fff640 10 monclient: _finish_hunting 0 2023-10-30T12:09:23.564 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.536+0000 7f9953fff640 1 monclient: found mon.noname-b 2023-10-30T12:09:23.564 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.536+0000 7f9953fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:23.564 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.536+0000 7f9953fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:09:23.565 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.536+0000 7f9953fff640 1 -- 172.21.15.92:0/179290651 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9954138fd0 con 0x7f995413e610 2023-10-30T12:09:23.565 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.538+0000 7f9953fff640 1 --2- 172.21.15.92:0/179290651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f995413e610 0x7f995413a9a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f994800b4d0 tx=0x7f994800b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:23.565 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.538+0000 7f99517fa640 10 client.?.objecter ms_handle_connect 0x7f995413e610 2023-10-30T12:09:23.565 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.538+0000 7f99517fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:23.567 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.541+0000 7f99517fa640 1 -- 172.21.15.92:0/179290651 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948010870 con 0x7f995413e610 2023-10-30T12:09:23.567 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f99517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:23.568 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f99517fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:23.568 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f99517fa640 10 monclient: dump: 2023-10-30T12:09:23.568 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:23.568 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:23.568 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.569 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.569 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:23.569 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:23.569 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:23.570 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:23.570 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:23.570 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:23.570 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f99517fa640 10 monclient: _finish_auth 0 2023-10-30T12:09:23.570 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f99517fa640 10 monclient: _check_auth_tickets 2023-10-30T12:09:23.571 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f99517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:23.571 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f99517fa640 1 -- 172.21.15.92:0/179290651 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99480109e0 con 0x7f995413e610 2023-10-30T12:09:23.571 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f995b1a9640 5 monclient: authenticate success, global_id 4141 2023-10-30T12:09:23.571 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f995b1a9640 10 monclient: _renew_subs 2023-10-30T12:09:23.572 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f995b1a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:23.572 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f995b1a9640 1 -- 172.21.15.92:0/179290651 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99541d7b20 con 0x7f995413e610 2023-10-30T12:09:23.572 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f99517fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:23.572 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f99517fa640 1 -- 172.21.15.92:0/179290651 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948010da0 con 0x7f995413e610 2023-10-30T12:09:23.572 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f99517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:23.573 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f99517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:23.573 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f99517fa640 10 monclient: dump: 2023-10-30T12:09:23.573 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:23.573 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:23.574 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.574 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:23.574 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:23.574 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:23.575 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:23.575 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:23.575 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:23.575 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:23.575 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f995b1a9640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:23.576 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f995b1a9640 10 monclient: _renew_subs 2023-10-30T12:09:23.576 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f995b1a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:23.576 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f9952ffd640 4 set_mon_vals no callback set 2023-10-30T12:09:23.576 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f995b1a9640 1 -- 172.21.15.92:0/179290651 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99541da600 con 0x7f995413e610 2023-10-30T12:09:23.576 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f995b1a9640 1 librados: init done 2023-10-30T12:09:23.577 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f995b1a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:23.577 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f995b1a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:23.577 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f995b1a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:23.577 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.542+0000 7f995b1a9640 1 -- 172.21.15.92:0/179290651 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99541d7e60 con 0x7f995413e610 2023-10-30T12:09:23.597 INFO:tasks.ceph.mgr.y.smithi003.stderr:2023-10-30T12:09:23.596+0000 7f0d0b844040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T12:09:23.638 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.636+0000 7f99517fa640 1 -- 172.21.15.92:0/179290651 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9948023780 con 0x7f995413e610 2023-10-30T12:09:23.638 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.636+0000 7f99517fa640 1 -- 172.21.15.92:0/179290651 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f994800f9b0 con 0x7f995413e610 2023-10-30T12:09:23.639 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.636+0000 7f99517fa640 10 client.4141.objecter ms_dispatch 0x7f9954000b90 osd_map(8..8 src has 1..8) v4 2023-10-30T12:09:23.639 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.636+0000 7f99517fa640 3 client.4141.objecter handle_osd_map got epochs [8,8] > 0 2023-10-30T12:09:23.639 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.636+0000 7f99517fa640 3 client.4141.objecter handle_osd_map decoding full epoch 8 2023-10-30T12:09:23.639 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.637+0000 7f99517fa640 20 client.4141.objecter dump_active .. 0 homeless 2023-10-30T12:09:23.639 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.637+0000 7f99517fa640 1 -- 172.21.15.92:0/179290651 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f994800fcb0 con 0x7f995413e610 2023-10-30T12:09:23.640 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.637+0000 7f99517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:23.640 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.637+0000 7f99517fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:23.729 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.728+0000 7f995b1a9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b39d2b0a-84ef-4909-bac7-47f146305a79", "id": 6}] 2023-10-30T12:09:23.730 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.728+0000 7f995b1a9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b39d2b0a-84ef-4909-bac7-47f146305a79", "id": 6}] 2023-10-30T12:09:23.730 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.728+0000 7f995b1a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:23.730 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.728+0000 7f995b1a9640 1 -- 172.21.15.92:0/179290651 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b39d2b0a-84ef-4909-bac7-47f146305a79", "id": 6} v 0) v1 -- 0x7f995404e9b0 con 0x7f995413e610 2023-10-30T12:09:23.798 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:23.796+0000 7f99517fa640 1 -- 172.21.15.92:0/179290651 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mgrmap(e 2) v1 ==== 44566+0+0 (secure 0 0 0) 0x7f9948023bc0 con 0x7f995413e610 2023-10-30T12:09:24.178 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.176+0000 7f99517fa640 1 -- 172.21.15.92:0/179290651 <== mon.1 v2:172.21.15.92:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b39d2b0a-84ef-4909-bac7-47f146305a79", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f994801f260 con 0x7f995413e610 2023-10-30T12:09:24.178 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.176+0000 7f99517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b39d2b0a-84ef-4909-bac7-47f146305a79", "id": 6}] 2023-10-30T12:09:24.179 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.176+0000 7f99517fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:24.179 INFO:teuthology.orchestra.run.smithi092.stdout:6 2023-10-30T12:09:24.181 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 10 librados: watch_flush enter 2023-10-30T12:09:24.182 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 10 librados: watch_flush exit 2023-10-30T12:09:24.182 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 20 client.4141.objecter shutdown clearing up homeless session... 2023-10-30T12:09:24.182 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 10 client.4141.objecter successfully canceled tick 2023-10-30T12:09:24.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 10 monclient: shutdown 2023-10-30T12:09:24.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:24.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 1 -- 172.21.15.92:0/179290651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f995413e610 msgr2=0x7f995413a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:24.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 1 --2- 172.21.15.92:0/179290651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f995413e610 0x7f995413a9a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f994800b4d0 tx=0x7f994800b990 comp rx=0 tx=0).stop 2023-10-30T12:09:24.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 1 -- 172.21.15.92:0/179290651 shutdown_connections 2023-10-30T12:09:24.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 1 --2- 172.21.15.92:0/179290651 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f99541327d0 0x7f995413aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:24.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 1 --2- 172.21.15.92:0/179290651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f995413e610 0x7f995413a9a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:24.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 1 --2- 172.21.15.92:0/179290651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9954071760 0x7f99541d75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:24.186 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 1 -- 172.21.15.92:0/179290651 >> 172.21.15.92:0/179290651 conn(0x7f9954069d50 msgr2=0x7f995413c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:24.186 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 1 -- 172.21.15.92:0/179290651 shutdown_connections 2023-10-30T12:09:24.187 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 1 -- 172.21.15.92:0/179290651 wait complete. 2023-10-30T12:09:24.187 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.177+0000 7f995b1a9640 1 librados: shutdown 2023-10-30T12:09:24.189 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 0a9d81ac-5a14-4fef-877a-982594787dd5 7 2023-10-30T12:09:24.329 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.327+0000 7f5b71a56640 10 monclient: get_monmap_and_config 2023-10-30T12:09:24.329 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.327+0000 7f5b71a56640 10 monclient: build_initial_monmap 2023-10-30T12:09:24.329 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.327+0000 7f5b71a56640 10 monclient: monmap: 2023-10-30T12:09:24.329 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:24.330 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:24.330 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:09:24.328095+0000 2023-10-30T12:09:24.330 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:09:24.328095+0000 2023-10-30T12:09:24.330 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:24.330 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:24.331 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:09:24.331 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:09:24.331 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:09:24.331 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:09:24.331 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:09:24.332 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:24.332 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.327+0000 7f5b71a56640 1 Processor -- start 2023-10-30T12:09:24.332 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.327+0000 7f5b71a56640 1 -- start start 2023-10-30T12:09:24.332 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.327+0000 7f5b71a56640 10 monclient: init 2023-10-30T12:09:24.334 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.328+0000 7f5b71a56640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:24.334 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.328+0000 7f5b71a56640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-10-30T12:09:24.335 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.328+0000 7f5b71a56640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b6c0b4440 addr v1:172.21.15.3:6789/0 2023-10-30T12:09:24.335 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.328+0000 7f5b71a56640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5b6c1455b0 0x7f5b6c13e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:24.335 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.328+0000 7f5b71a56640 10 monclient(hunting): picked mon.noname-a con 0x7f5b6c1455b0 addr v2:172.21.15.3:3300/0 2023-10-30T12:09:24.335 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.328+0000 7f5b71a56640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5b6c13eb80 0x7f5b6c13efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:24.335 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.328+0000 7f5b71a56640 10 monclient(hunting): picked mon.noname-c con 0x7f5b6c13eb80 addr v2:172.21.15.92:3300/0 2023-10-30T12:09:24.335 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.328+0000 7f5b71a56640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b6c0a9da0 con 0x7f5b6c0b4440 2023-10-30T12:09:24.336 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.328+0000 7f5b71a56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:24.336 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.328+0000 7f5b71a56640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6c067a10 con 0x7f5b6c1455b0 2023-10-30T12:09:24.336 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.328+0000 7f5b71a56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:24.336 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.328+0000 7f5b71a56640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6c0aa080 con 0x7f5b6c13eb80 2023-10-30T12:09:24.336 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.328+0000 7f5b71a56640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:24.337 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.328+0000 7f5b71a56640 10 monclient(hunting): authenticate will time out at 1361.505729s 2023-10-30T12:09:24.337 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.328+0000 7f5b6b7fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5b6c13eb80 0x7f5b6c13efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:24.337 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.329+0000 7f5b6b7fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5b6c13eb80 0x7f5b6c13efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:34434/0 (socket says 172.21.15.92:34434) 2023-10-30T12:09:24.337 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.329+0000 7f5b6b7fe640 1 -- 172.21.15.92:0/1004351650 learned_addr learned my addr 172.21.15.92:0/1004351650 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-10-30T12:09:24.338 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.329+0000 7f5b6b7fe640 10 monclient(hunting): get_auth_request con 0x7f5b6c13eb80 auth_method 0 2023-10-30T12:09:24.338 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.329+0000 7f5b6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:24.338 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.329+0000 7f5b6b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:24.338 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.329+0000 7f5b6b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:24.338 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.330+0000 7f5b6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:24.339 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.330+0000 7f5b6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:24.339 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.330+0000 7f5b6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:24.339 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.331+0000 7f5b6b7fe640 10 monclient(hunting): handle_auth_done global_id 4147 payload 274 2023-10-30T12:09:24.339 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.331+0000 7f5b6b7fe640 1 -- 172.21.15.92:0/1004351650 >> v2:172.21.15.3:3300/0 conn(0x7f5b6c1455b0 msgr2=0x7f5b6c13e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:24.339 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.331+0000 7f5b6b7fe640 1 --2- 172.21.15.92:0/1004351650 >> v2:172.21.15.3:3300/0 conn(0x7f5b6c1455b0 0x7f5b6c13e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:24.340 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.331+0000 7f5b6b7fe640 1 -- 172.21.15.92:0/1004351650 >> v1:172.21.15.3:6789/0 conn(0x7f5b6c0b4440 legacy=0x7f5b6c144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:24.340 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.331+0000 7f5b6b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:09:24.340 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.331+0000 7f5b6b7fe640 1 monclient: found mon.noname-c 2023-10-30T12:09:24.340 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.331+0000 7f5b6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:24.340 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.331+0000 7f5b6b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:09:24.340 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.331+0000 7f5b6b7fe640 1 -- 172.21.15.92:0/1004351650 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c0aa360 con 0x7f5b6c13eb80 2023-10-30T12:09:24.341 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.333+0000 7f5b6b7fe640 1 --2- 172.21.15.92:0/1004351650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b6c13eb80 0x7f5b6c13efd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0b1780 tx=0x7f5b6402bdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb03a1cb4adb68e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:24.344 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b69ffb640 1 -- 172.21.15.92:0/1004351650 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6402c960 con 0x7f5b6c13eb80 2023-10-30T12:09:24.344 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:24.345 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b69ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:24.345 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b69ffb640 10 monclient: dump: 2023-10-30T12:09:24.345 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:24.345 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:24.345 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:24.346 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:24.346 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:24.346 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:24.346 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:24.346 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:24.347 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:24.347 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:24.347 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b69ffb640 10 monclient: _finish_auth 0 2023-10-30T12:09:24.347 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b69ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:09:24.347 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:24.348 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b69ffb640 1 -- 172.21.15.92:0/1004351650 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6402cad0 con 0x7f5b6c13eb80 2023-10-30T12:09:24.348 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b71a56640 5 monclient: authenticate success, global_id 4147 2023-10-30T12:09:24.348 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b71a56640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:24.348 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b69ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:24.348 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b71a56640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:24.349 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b69ffb640 1 -- 172.21.15.92:0/1004351650 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6402ce70 con 0x7f5b6c13eb80 2023-10-30T12:09:24.349 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b71a56640 4 set_mon_vals no callback set 2023-10-30T12:09:24.349 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:24.349 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b69ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:24.349 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.336+0000 7f5b69ffb640 10 monclient: dump: 2023-10-30T12:09:24.350 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:24.350 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:24.350 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:24.350 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:24.350 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:24.351 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:24.351 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:24.351 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:24.351 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:24.351 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:24.352 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.337+0000 7f5b71a56640 10 monclient: shutdown 2023-10-30T12:09:24.352 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.337+0000 7f5b71a56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:24.352 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.337+0000 7f5b71a56640 1 -- 172.21.15.92:0/1004351650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b6c13eb80 msgr2=0x7f5b6c13efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:24.352 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.337+0000 7f5b71a56640 1 --2- 172.21.15.92:0/1004351650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b6c13eb80 0x7f5b6c13efd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0b1780 tx=0x7f5b6402bdc0 comp rx=0 tx=0).stop 2023-10-30T12:09:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.337+0000 7f5b71a56640 1 -- 172.21.15.92:0/1004351650 shutdown_connections 2023-10-30T12:09:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.337+0000 7f5b71a56640 1 --2- 172.21.15.92:0/1004351650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b6c13eb80 0x7f5b6c13efd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.337+0000 7f5b71a56640 1 --2- 172.21.15.92:0/1004351650 >> v2:172.21.15.3:3300/0 conn(0x7f5b6c1455b0 0x7f5b6c13e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.337+0000 7f5b71a56640 1 -- 172.21.15.92:0/1004351650 >> 172.21.15.92:0/1004351650 conn(0x7f5b6c065ea0 msgr2=0x7f5b6c0662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.337+0000 7f5b71a56640 1 -- 172.21.15.92:0/1004351650 shutdown_connections 2023-10-30T12:09:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.337+0000 7f5b71a56640 1 -- 172.21.15.92:0/1004351650 wait complete. 2023-10-30T12:09:24.354 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.338+0000 7f5b71a56640 10 monclient: build_initial_monmap 2023-10-30T12:09:24.354 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.338+0000 7f5b71a56640 10 monclient: monmap: 2023-10-30T12:09:24.354 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-10-30T12:09:24.354 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:24.354 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-10-30T12:09:24.355 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-10-30T12:09:24.355 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:24.355 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-10-30T12:09:24.355 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:09:24.355 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:09:24.355 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:09:24.356 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:24.356 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.338+0000 7f5b71a56640 1 librados: starting msgr at 2023-10-30T12:09:24.356 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.338+0000 7f5b71a56640 1 librados: starting objecter 2023-10-30T12:09:24.356 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.338+0000 7f5b71a56640 1 Processor -- start 2023-10-30T12:09:24.356 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.338+0000 7f5b71a56640 1 -- start start 2023-10-30T12:09:24.357 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.338+0000 7f5b71a56640 1 librados: setting wanted keys 2023-10-30T12:09:24.357 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.338+0000 7f5b71a56640 1 librados: calling monclient init 2023-10-30T12:09:24.357 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.338+0000 7f5b71a56640 10 monclient: init 2023-10-30T12:09:24.357 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:24.357 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:09:24.358 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b6c0b4440 0x7f5b6c13b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:24.358 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 10 monclient(hunting): picked mon.noname-b con 0x7f5b6c0b4440 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:24.358 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b6c13eb80 0x7f5b6c1da250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:24.358 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 10 monclient(hunting): picked mon.noname-c con 0x7f5b6c13eb80 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:24.358 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6c1455b0 0x7f5b6c1cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:24.359 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 10 monclient(hunting): picked mon.noname-a con 0x7f5b6c1455b0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:24.359 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:24.359 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c0a8e70 con 0x7f5b6c1455b0 2023-10-30T12:09:24.359 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:24.359 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c13b030 con 0x7f5b6c0b4440 2023-10-30T12:09:24.359 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:24.360 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c0b1190 con 0x7f5b6c13eb80 2023-10-30T12:09:24.360 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:24.360 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b71a56640 10 monclient(hunting): authenticate will time out at 1361.516729s 2023-10-30T12:09:24.360 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.339+0000 7f5b6affd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b6c0b4440 0x7f5b6c13b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:24.360 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6a7fc640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b6c13eb80 0x7f5b6c1da250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:24.360 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6affd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b6c0b4440 0x7f5b6c13b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:34448/0 (socket says 172.21.15.92:34448) 2023-10-30T12:09:24.361 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6affd640 1 -- 172.21.15.92:0/1254458156 learned_addr learned my addr 172.21.15.92:0/1254458156 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-10-30T12:09:24.361 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6affd640 10 monclient(hunting): get_auth_request con 0x7f5b6c0b4440 auth_method 0 2023-10-30T12:09:24.361 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:24.361 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:24.362 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:24.362 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6a7fc640 10 monclient(hunting): get_auth_request con 0x7f5b6c13eb80 auth_method 0 2023-10-30T12:09:24.362 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:24.362 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:24.362 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:24.363 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:24.363 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:24.363 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:24.363 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:24.363 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:24.364 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.340+0000 7f5b6a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:24.364 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.343+0000 7f5b6affd640 10 monclient(hunting): handle_auth_done global_id 4150 payload 824 2023-10-30T12:09:24.364 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.343+0000 7f5b6affd640 1 -- 172.21.15.92:0/1254458156 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b6c13eb80 msgr2=0x7f5b6c1da250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:24.364 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.343+0000 7f5b6affd640 1 --2- 172.21.15.92:0/1254458156 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b6c13eb80 0x7f5b6c1da250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:24.364 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.343+0000 7f5b6affd640 1 -- 172.21.15.92:0/1254458156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6c1455b0 msgr2=0x7f5b6c1cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:24.365 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.343+0000 7f5b6affd640 1 --2- 172.21.15.92:0/1254458156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6c1455b0 0x7f5b6c1cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:24.365 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.343+0000 7f5b6affd640 10 monclient: _finish_hunting 0 2023-10-30T12:09:24.365 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.343+0000 7f5b6affd640 1 monclient: found mon.noname-b 2023-10-30T12:09:24.365 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.343+0000 7f5b6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:24.365 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.343+0000 7f5b6affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:09:24.366 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.343+0000 7f5b6affd640 1 -- 172.21.15.92:0/1254458156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c1da790 con 0x7f5b6c0b4440 2023-10-30T12:09:24.367 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.345+0000 7f5b6affd640 1 --2- 172.21.15.92:0/1254458156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b6c0b4440 0x7f5b6c13b1c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c00c440 tx=0x7f5b5c00c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:24.367 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.345+0000 7f5b53fff640 10 client.?.objecter ms_handle_connect 0x7f5b6c0b4440 2023-10-30T12:09:24.367 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.345+0000 7f5b53fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:24.368 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b53fff640 1 -- 172.21.15.92:0/1254458156 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c005480 con 0x7f5b6c0b4440 2023-10-30T12:09:24.368 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:24.368 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b53fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:24.368 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b53fff640 10 monclient: dump: 2023-10-30T12:09:24.369 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:24.369 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:24.369 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:24.369 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:24.369 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:24.370 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:24.370 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:24.370 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:24.370 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:24.370 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:24.371 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b53fff640 10 monclient: _finish_auth 0 2023-10-30T12:09:24.371 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b53fff640 10 monclient: _check_auth_tickets 2023-10-30T12:09:24.371 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:24.371 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b53fff640 1 -- 172.21.15.92:0/1254458156 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5c00e040 con 0x7f5b6c0b4440 2023-10-30T12:09:24.371 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b71a56640 5 monclient: authenticate success, global_id 4150 2023-10-30T12:09:24.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b71a56640 10 monclient: _renew_subs 2023-10-30T12:09:24.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b71a56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:24.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b71a56640 1 -- 172.21.15.92:0/1254458156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6c1daa50 con 0x7f5b6c0b4440 2023-10-30T12:09:24.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b53fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:24.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b53fff640 1 -- 172.21.15.92:0/1254458156 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c003830 con 0x7f5b6c0b4440 2023-10-30T12:09:24.373 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:24.373 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b53fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:24.373 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b53fff640 10 monclient: dump: 2023-10-30T12:09:24.373 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-10-30T12:09:24.373 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:24.373 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:24.374 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:24.374 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:24.374 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 3 2023-10-30T12:09:24.374 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:24.374 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:24.375 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:24.375 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:09:24.375 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b697fa640 4 set_mon_vals no callback set 2023-10-30T12:09:24.375 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b71a56640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:24.375 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b71a56640 10 monclient: _renew_subs 2023-10-30T12:09:24.376 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b71a56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:24.376 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b71a56640 1 -- 172.21.15.92:0/1254458156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6c1cdfd0 con 0x7f5b6c0b4440 2023-10-30T12:09:24.376 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.348+0000 7f5b71a56640 1 librados: init done 2023-10-30T12:09:24.376 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.349+0000 7f5b71a56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:24.376 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.349+0000 7f5b71a56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:24.376 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.349+0000 7f5b71a56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:24.377 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.349+0000 7f5b71a56640 1 -- 172.21.15.92:0/1254458156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b6c0a9410 con 0x7f5b6c0b4440 2023-10-30T12:09:24.377 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.360+0000 7f5b53fff640 1 -- 172.21.15.92:0/1254458156 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 2) v1 ==== 44566+0+0 (secure 0 0 0) 0x7f5b5c01b3b0 con 0x7f5b6c0b4440 2023-10-30T12:09:24.459 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.458+0000 7f5b53fff640 1 -- 172.21.15.92:0/1254458156 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f5b5c01a760 con 0x7f5b6c0b4440 2023-10-30T12:09:24.460 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.458+0000 7f5b53fff640 10 client.4150.objecter ms_dispatch 0x7f5b6c000b90 osd_map(9..9 src has 1..9) v4 2023-10-30T12:09:24.460 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.458+0000 7f5b53fff640 3 client.4150.objecter handle_osd_map got epochs [9,9] > 0 2023-10-30T12:09:24.460 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.458+0000 7f5b53fff640 3 client.4150.objecter handle_osd_map decoding full epoch 9 2023-10-30T12:09:24.460 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.458+0000 7f5b53fff640 20 client.4150.objecter dump_active .. 0 homeless 2023-10-30T12:09:24.461 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.458+0000 7f5b53fff640 1 -- 172.21.15.92:0/1254458156 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f5b5c01a9c0 con 0x7f5b6c0b4440 2023-10-30T12:09:24.461 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.458+0000 7f5b53fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:24.461 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.458+0000 7f5b53fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:24.528 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.526+0000 7f5b71a56640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0a9d81ac-5a14-4fef-877a-982594787dd5", "id": 7}] 2023-10-30T12:09:24.528 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.526+0000 7f5b71a56640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0a9d81ac-5a14-4fef-877a-982594787dd5", "id": 7}] 2023-10-30T12:09:24.528 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.526+0000 7f5b71a56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:24.529 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.526+0000 7f5b71a56640 1 -- 172.21.15.92:0/1254458156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0a9d81ac-5a14-4fef-877a-982594787dd5", "id": 7} v 0) v1 -- 0x7f5b6c1da790 con 0x7f5b6c0b4440 2023-10-30T12:09:24.948 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.947+0000 7f5b53fff640 1 -- 172.21.15.92:0/1254458156 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0a9d81ac-5a14-4fef-877a-982594787dd5", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5b5c019d70 con 0x7f5b6c0b4440 2023-10-30T12:09:24.949 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.947+0000 7f5b53fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0a9d81ac-5a14-4fef-877a-982594787dd5", "id": 7}] 2023-10-30T12:09:24.949 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.947+0000 7f5b53fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:24.949 INFO:teuthology.orchestra.run.smithi092.stdout:7 2023-10-30T12:09:24.950 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 10 librados: watch_flush enter 2023-10-30T12:09:24.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 10 librados: watch_flush exit 2023-10-30T12:09:24.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 20 client.4150.objecter shutdown clearing up homeless session... 2023-10-30T12:09:24.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 10 client.4150.objecter successfully canceled tick 2023-10-30T12:09:24.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 10 monclient: shutdown 2023-10-30T12:09:24.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:24.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 1 -- 172.21.15.92:0/1254458156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b6c0b4440 msgr2=0x7f5b6c13b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:24.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 1 --2- 172.21.15.92:0/1254458156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b6c0b4440 0x7f5b6c13b1c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c00c440 tx=0x7f5b5c00c900 comp rx=0 tx=0).stop 2023-10-30T12:09:24.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 1 -- 172.21.15.92:0/1254458156 shutdown_connections 2023-10-30T12:09:24.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 1 --2- 172.21.15.92:0/1254458156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6c1455b0 0x7f5b6c1cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:24.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 1 --2- 172.21.15.92:0/1254458156 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b6c13eb80 0x7f5b6c1da250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:24.953 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 1 --2- 172.21.15.92:0/1254458156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b6c0b4440 0x7f5b6c13b1c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:24.953 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 1 -- 172.21.15.92:0/1254458156 >> 172.21.15.92:0/1254458156 conn(0x7f5b6c065ea0 msgr2=0x7f5b6c13cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:24.953 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 1 -- 172.21.15.92:0/1254458156 shutdown_connections 2023-10-30T12:09:24.953 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 1 -- 172.21.15.92:0/1254458156 wait complete. 2023-10-30T12:09:24.953 INFO:teuthology.orchestra.run.smithi092.stderr:2023-10-30T12:09:24.948+0000 7f5b71a56640 1 librados: shutdown 2023-10-30T12:09:24.959 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:09:24.959 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-30T12:09:24.960 DEBUG:teuthology.orchestra.run.smithi003:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-10-30T12:09:24.962 INFO:tasks.ceph.osd.0:Started 2023-10-30T12:09:24.963 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:09:24.963 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-30T12:09:24.963 DEBUG:teuthology.orchestra.run.smithi003:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-10-30T12:09:24.966 INFO:tasks.ceph.osd.1:Started 2023-10-30T12:09:24.966 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:09:24.966 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-30T12:09:24.967 DEBUG:teuthology.orchestra.run.smithi003:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-10-30T12:09:24.969 INFO:tasks.ceph.osd.2:Started 2023-10-30T12:09:24.969 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:09:24.969 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-30T12:09:24.970 DEBUG:teuthology.orchestra.run.smithi003:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-10-30T12:09:24.972 INFO:tasks.ceph.osd.3:Started 2023-10-30T12:09:24.972 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:09:24.973 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-30T12:09:24.973 DEBUG:teuthology.orchestra.run.smithi092:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-10-30T12:09:24.975 INFO:tasks.ceph.osd.4:Started 2023-10-30T12:09:24.976 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:09:24.976 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-30T12:09:24.976 DEBUG:teuthology.orchestra.run.smithi092:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-10-30T12:09:24.978 INFO:tasks.ceph.osd.5:Started 2023-10-30T12:09:24.979 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:09:24.979 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-30T12:09:24.979 DEBUG:teuthology.orchestra.run.smithi092:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-10-30T12:09:24.981 INFO:tasks.ceph.osd.6:Started 2023-10-30T12:09:24.982 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T12:09:24.982 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-30T12:09:24.982 DEBUG:teuthology.orchestra.run.smithi092:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-10-30T12:09:24.985 INFO:tasks.ceph.osd.7:Started 2023-10-30T12:09:24.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:09:25.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.138+0000 7f1ecd173640 10 monclient: get_monmap_and_config 2023-10-30T12:09:25.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.138+0000 7f1ecd173640 10 monclient: build_initial_monmap 2023-10-30T12:09:25.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.138+0000 7f1ecd173640 10 monclient: monmap: 2023-10-30T12:09:25.140 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:09:25.140 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:25.141 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:09:25.139267+0000 2023-10-30T12:09:25.141 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:09:25.139267+0000 2023-10-30T12:09:25.141 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:25.141 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:25.141 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:09:25.142 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:09:25.142 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:09:25.142 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:09:25.142 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:09:25.142 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:09:25.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.139+0000 7f1ecd173640 1 Processor -- start 2023-10-30T12:09:25.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.139+0000 7f1ecd173640 1 -- start start 2023-10-30T12:09:25.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.139+0000 7f1ecd173640 10 monclient: init 2023-10-30T12:09:25.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.140+0000 7f1ecd173640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:25.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.140+0000 7f1ecd173640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-10-30T12:09:25.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.140+0000 7f1ecd173640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1ec80b4390 0x7f1ec80a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:25.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.140+0000 7f1ecd173640 10 monclient(hunting): picked mon.noname-c con 0x7f1ec80b4390 addr v2:172.21.15.92:3300/0 2023-10-30T12:09:25.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.140+0000 7f1ecd173640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ec81418a0 addr v1:172.21.15.92:6789/0 2023-10-30T12:09:25.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.140+0000 7f1ecd173640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1ec8141ce0 0x7f1ec80b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:25.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.140+0000 7f1ecd173640 10 monclient(hunting): picked mon.noname-a con 0x7f1ec8141ce0 addr v2:172.21.15.3:3300/0 2023-10-30T12:09:25.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.141+0000 7f1ecd173640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec80a8f60 con 0x7f1ec81418a0 2023-10-30T12:09:25.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.141+0000 7f1ecd173640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:25.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.141+0000 7f1ecd173640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec8067a10 con 0x7f1ec8141ce0 2023-10-30T12:09:25.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.141+0000 7f1ecd173640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:25.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.141+0000 7f1ecd173640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec80b4770 con 0x7f1ec80b4390 2023-10-30T12:09:25.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.141+0000 7f1ecd173640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:25.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.141+0000 7f1ecd173640 10 monclient(hunting): authenticate will time out at 1356.355444s 2023-10-30T12:09:25.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.170+0000 7f1ec759e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1ec8141ce0 0x7f1ec80b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:25.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.170+0000 7f1ec6d9d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1ec80b4390 0x7f1ec80a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:25.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.171+0000 7f1ec6d9d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1ec80b4390 0x7f1ec80a8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50304/0 (socket says 172.21.15.3:50304) 2023-10-30T12:09:25.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.171+0000 7f1ec6d9d640 1 -- 172.21.15.3:0/3035832159 learned_addr learned my addr 172.21.15.3:0/3035832159 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:09:25.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.171+0000 7f1ec6d9d640 10 monclient(hunting): get_auth_request con 0x7f1ec80b4390 auth_method 0 2023-10-30T12:09:25.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.171+0000 7f1ec6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:25.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.171+0000 7f1ec6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:25.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.171+0000 7f1ec6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:25.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.172+0000 7f1ec6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:25.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.172+0000 7f1ec6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:25.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.172+0000 7f1ec6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:25.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.173+0000 7f1ec759e640 10 monclient(hunting): get_auth_request con 0x7f1ec8141ce0 auth_method 0 2023-10-30T12:09:25.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.173+0000 7f1ec759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:25.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.173+0000 7f1ec759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:25.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.173+0000 7f1ec759e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:25.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.174+0000 7f1ec759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:25.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.174+0000 7f1ec759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:25.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.174+0000 7f1ec759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:25.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.175+0000 7f1ec6d9d640 10 monclient(hunting): handle_auth_done global_id 4153 payload 274 2023-10-30T12:09:25.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.175+0000 7f1ec6d9d640 1 -- 172.21.15.3:0/3035832159 >> v2:172.21.15.3:3300/0 conn(0x7f1ec8141ce0 msgr2=0x7f1ec80b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:25.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.175+0000 7f1ec6d9d640 1 --2- 172.21.15.3:0/3035832159 >> v2:172.21.15.3:3300/0 conn(0x7f1ec8141ce0 0x7f1ec80b2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:25.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.175+0000 7f1ec6d9d640 1 -- 172.21.15.3:0/3035832159 >> v1:172.21.15.92:6789/0 conn(0x7f1ec81418a0 legacy=0x7f1ec80b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.175+0000 7f1ec6d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:09:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.175+0000 7f1ec6d9d640 1 monclient: found mon.noname-c 2023-10-30T12:09:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.175+0000 7f1ec6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.175+0000 7f1ec6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:09:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.175+0000 7f1ec6d9d640 1 -- 172.21.15.3:0/3035832159 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec80aa3d0 con 0x7f1ec80b4390 2023-10-30T12:09:25.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.179+0000 7f1ec6d9d640 1 --2- 172.21.15.3:0/3035832159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ec80b4390 0x7f1ec80a8770 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80b4a50 tx=0x7f1eb802b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8aad2ce5340a82fc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:25.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.184+0000 7f1ec5d9b640 1 -- 172.21.15.3:0/3035832159 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8031890 con 0x7f1ec80b4390 2023-10-30T12:09:25.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.184+0000 7f1ec5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:25.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.184+0000 7f1ec5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:25.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.184+0000 7f1ec5d9b640 10 monclient: dump: 2023-10-30T12:09:25.187 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:09:25.187 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:25.187 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:25.187 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:25.187 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:25.188 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:25.188 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:25.188 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:25.188 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:25.188 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:09:25.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.184+0000 7f1ec5d9b640 10 monclient: _finish_auth 0 2023-10-30T12:09:25.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.184+0000 7f1ec5d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:09:25.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.184+0000 7f1ec5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:25.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.184+0000 7f1ec5d9b640 1 -- 172.21.15.3:0/3035832159 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb8031a00 con 0x7f1ec80b4390 2023-10-30T12:09:25.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.184+0000 7f1ec5d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:25.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.184+0000 7f1ec5d9b640 1 -- 172.21.15.3:0/3035832159 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8031de0 con 0x7f1ec80b4390 2023-10-30T12:09:25.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.184+0000 7f1ec5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:25.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.184+0000 7f1ec5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:25.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.184+0000 7f1ec5d9b640 10 monclient: dump: 2023-10-30T12:09:25.191 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:09:25.191 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:25.191 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:25.191 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:25.191 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:25.192 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:25.192 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:25.192 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:25.192 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:25.192 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:09:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.185+0000 7f1ecd173640 5 monclient: authenticate success, global_id 4153 2023-10-30T12:09:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.185+0000 7f1ecd173640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.185+0000 7f1ecd173640 4 set_mon_vals no callback set 2023-10-30T12:09:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.186+0000 7f1ecd173640 10 monclient: shutdown 2023-10-30T12:09:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.186+0000 7f1ecd173640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:25.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.186+0000 7f1ecd173640 1 -- 172.21.15.3:0/3035832159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ec80b4390 msgr2=0x7f1ec80a8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:25.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.186+0000 7f1ecd173640 1 --2- 172.21.15.3:0/3035832159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ec80b4390 0x7f1ec80a8770 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80b4a50 tx=0x7f1eb802b930 comp rx=0 tx=0).stop 2023-10-30T12:09:25.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.186+0000 7f1ecd173640 1 -- 172.21.15.3:0/3035832159 shutdown_connections 2023-10-30T12:09:25.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.186+0000 7f1ecd173640 1 --2- 172.21.15.3:0/3035832159 >> v2:172.21.15.3:3300/0 conn(0x7f1ec8141ce0 0x7f1ec80b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:25.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.186+0000 7f1ecd173640 1 --2- 172.21.15.3:0/3035832159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ec80b4390 0x7f1ec80a8770 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:25.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.186+0000 7f1ecd173640 1 -- 172.21.15.3:0/3035832159 >> 172.21.15.3:0/3035832159 conn(0x7f1ec8065ea0 msgr2=0x7f1ec80662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:25.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.186+0000 7f1ecd173640 1 -- 172.21.15.3:0/3035832159 shutdown_connections 2023-10-30T12:09:25.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.186+0000 7f1ecd173640 1 -- 172.21.15.3:0/3035832159 wait complete. 2023-10-30T12:09:25.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.187+0000 7f1ecd173640 10 monclient: build_initial_monmap 2023-10-30T12:09:25.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.188+0000 7f1ecd173640 10 monclient: monmap: 2023-10-30T12:09:25.199 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:09:25.199 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:25.199 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:09:25.199 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:09:25.200 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:25.200 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:09:25.200 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:09:25.200 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:09:25.201 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:09:25.201 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:09:25.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.188+0000 7f1ecd173640 1 librados: starting msgr at 2023-10-30T12:09:25.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.188+0000 7f1ecd173640 1 librados: starting objecter 2023-10-30T12:09:25.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.189+0000 7f1ecd173640 1 Processor -- start 2023-10-30T12:09:25.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.189+0000 7f1ecd173640 1 -- start start 2023-10-30T12:09:25.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.189+0000 7f1ecd173640 1 librados: setting wanted keys 2023-10-30T12:09:25.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.189+0000 7f1ecd173640 1 librados: calling monclient init 2023-10-30T12:09:25.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.189+0000 7f1ecd173640 10 monclient: init 2023-10-30T12:09:25.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.190+0000 7f1ecd173640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:25.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.190+0000 7f1ecd173640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:09:25.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.190+0000 7f1ecd173640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ec80b4390 0x7f1ec81da4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:25.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.190+0000 7f1ecd173640 10 monclient(hunting): picked mon.noname-b con 0x7f1ec80b4390 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:25.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.190+0000 7f1ecd173640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1ec81418a0 0x7f1ec8135270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:25.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.191+0000 7f1ecd173640 10 monclient(hunting): picked mon.noname-c con 0x7f1ec81418a0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:25.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.191+0000 7f1ecd173640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ec8141ce0 0x7f1ec81357b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:25.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.191+0000 7f1ecd173640 10 monclient(hunting): picked mon.noname-a con 0x7f1ec8141ce0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:25.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.191+0000 7f1ecd173640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:25.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.191+0000 7f1ecd173640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec8142390 con 0x7f1ec8141ce0 2023-10-30T12:09:25.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.191+0000 7f1ecd173640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:25.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.191+0000 7f1ecd173640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec81dae90 con 0x7f1ec80b4390 2023-10-30T12:09:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.191+0000 7f1ecd173640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.191+0000 7f1ecd173640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec8142d30 con 0x7f1ec81418a0 2023-10-30T12:09:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.191+0000 7f1ec6d9d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ec80b4390 0x7f1ec81da4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.191+0000 7f1ecd173640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:25.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.192+0000 7f1ec6d9d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ec80b4390 0x7f1ec81da4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50308/0 (socket says 172.21.15.3:50308) 2023-10-30T12:09:25.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.192+0000 7f1ec6d9d640 1 -- 172.21.15.3:0/2308942472 learned_addr learned my addr 172.21.15.3:0/2308942472 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:09:25.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.192+0000 7f1ecd173640 10 monclient(hunting): authenticate will time out at 1356.406586s 2023-10-30T12:09:25.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.192+0000 7f1ec6d9d640 10 monclient(hunting): get_auth_request con 0x7f1ec80b4390 auth_method 0 2023-10-30T12:09:25.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.192+0000 7f1ec6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:25.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.192+0000 7f1ec6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:25.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.192+0000 7f1ec6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:25.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.192+0000 7f1ec759e640 1 --2- 172.21.15.3:0/2308942472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ec8141ce0 0x7f1ec81357b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:25.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.193+0000 7f1ec6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:25.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.193+0000 7f1ec6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:25.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.193+0000 7f1ec6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:25.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.193+0000 7f1ec759e640 10 monclient(hunting): get_auth_request con 0x7f1ec8141ce0 auth_method 0 2023-10-30T12:09:25.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.193+0000 7f1ec759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:25.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.193+0000 7f1ec759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:25.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.193+0000 7f1ec759e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:25.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.194+0000 7f1ec759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:25.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.194+0000 7f1ec759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:25.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.194+0000 7f1ec759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:25.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.197+0000 7f1ec6d9d640 10 monclient(hunting): handle_auth_done global_id 4156 payload 824 2023-10-30T12:09:25.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.197+0000 7f1ec6d9d640 1 -- 172.21.15.3:0/2308942472 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1ec81418a0 msgr2=0x7f1ec8135270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:25.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.197+0000 7f1ec6d9d640 1 --2- 172.21.15.3:0/2308942472 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1ec81418a0 0x7f1ec8135270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:25.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.197+0000 7f1ec6d9d640 1 -- 172.21.15.3:0/2308942472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ec8141ce0 msgr2=0x7f1ec81357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:25.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.197+0000 7f1ec6d9d640 1 --2- 172.21.15.3:0/2308942472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ec8141ce0 0x7f1ec81357b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:25.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.197+0000 7f1ec6d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:09:25.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.197+0000 7f1ec6d9d640 1 monclient: found mon.noname-b 2023-10-30T12:09:25.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.197+0000 7f1ec6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:25.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.197+0000 7f1ec6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:09:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.197+0000 7f1ec6d9d640 1 -- 172.21.15.3:0/2308942472 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec8135cf0 con 0x7f1ec80b4390 2023-10-30T12:09:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.200+0000 7f1ec6d9d640 1 --2- 172.21.15.3:0/2308942472 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ec80b4390 0x7f1ec81da4c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8000c00 tx=0x7f1eb80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.200+0000 7f1eaf7fe640 10 client.?.objecter ms_handle_connect 0x7f1ec80b4390 2023-10-30T12:09:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.200+0000 7f1eaf7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.205+0000 7f1eaf7fe640 1 -- 172.21.15.3:0/2308942472 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8047070 con 0x7f1ec80b4390 2023-10-30T12:09:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.205+0000 7f1eaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.205+0000 7f1eaf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.205+0000 7f1eaf7fe640 10 monclient: dump: 2023-10-30T12:09:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:09:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:25.219 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:25.219 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:25.219 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:25.219 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:25.219 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:25.220 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:09:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.205+0000 7f1eaf7fe640 10 monclient: _finish_auth 0 2023-10-30T12:09:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.205+0000 7f1eaf7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:09:25.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.205+0000 7f1eaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:25.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.205+0000 7f1eaf7fe640 1 -- 172.21.15.3:0/2308942472 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb8043d40 con 0x7f1ec80b4390 2023-10-30T12:09:25.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.205+0000 7f1eaf7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:25.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.205+0000 7f1ec559a640 4 set_mon_vals no callback set 2023-10-30T12:09:25.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.206+0000 7f1ecd173640 5 monclient: authenticate success, global_id 4156 2023-10-30T12:09:25.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.206+0000 7f1eaf7fe640 1 -- 172.21.15.3:0/2308942472 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb803aa10 con 0x7f1ec80b4390 2023-10-30T12:09:25.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.206+0000 7f1ecd173640 10 monclient: _renew_subs 2023-10-30T12:09:25.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.206+0000 7f1ecd173640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:25.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.206+0000 7f1ecd173640 1 -- 172.21.15.3:0/2308942472 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec81daa00 con 0x7f1ec80b4390 2023-10-30T12:09:25.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.206+0000 7f1eaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:25.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.206+0000 7f1eaf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:25.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.206+0000 7f1eaf7fe640 10 monclient: dump: 2023-10-30T12:09:25.223 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:09:25.223 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:25.223 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:25.224 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:25.224 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:25.224 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:25.224 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:25.224 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:25.225 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:25.225 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:09:25.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.206+0000 7f1ecd173640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:25.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.207+0000 7f1ecd173640 10 monclient: _renew_subs 2023-10-30T12:09:25.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.207+0000 7f1ecd173640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:25.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.207+0000 7f1ecd173640 1 -- 172.21.15.3:0/2308942472 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec80b0440 con 0x7f1ec80b4390 2023-10-30T12:09:25.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.207+0000 7f1ecd173640 1 librados: init done 2023-10-30T12:09:25.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.208+0000 7f1ead7fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:25.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.208+0000 7f1ead7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:25.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.208+0000 7f1ead7fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:25.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.208+0000 7f1ead7fa640 1 -- 172.21.15.3:0/2308942472 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea0005040 con 0x7f1ec80b4390 2023-10-30T12:09:25.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.392+0000 7f1eaf7fe640 1 -- 172.21.15.3:0/2308942472 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 2) v1 ==== 44566+0+0 (secure 0 0 0) 0x7f1eb803ab80 con 0x7f1ec80b4390 2023-10-30T12:09:25.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.408+0000 7f1eaf7fe640 1 -- 172.21.15.3:0/2308942472 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f1eb8039cb0 con 0x7f1ec80b4390 2023-10-30T12:09:25.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.408+0000 7f1eaf7fe640 10 client.4156.objecter ms_dispatch 0x7f1ec8000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T12:09:25.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.408+0000 7f1eaf7fe640 3 client.4156.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T12:09:25.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.408+0000 7f1eaf7fe640 3 client.4156.objecter handle_osd_map decoding full epoch 10 2023-10-30T12:09:25.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.408+0000 7f1eaf7fe640 20 client.4156.objecter dump_active .. 0 homeless 2023-10-30T12:09:25.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.417+0000 7f1eaf7fe640 1 -- 172.21.15.3:0/2308942472 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f1eb804cbd0 con 0x7f1ec80b4390 2023-10-30T12:09:25.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.417+0000 7f1eaf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:25.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.417+0000 7f1eaf7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:25.467 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:25.465+0000 7f679bdff640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-30T12:09:25.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.534+0000 7f1ead7fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:25.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.534+0000 7f1ead7fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:25.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.534+0000 7f1ead7fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:25.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.534+0000 7f1ead7fa640 1 -- 172.21.15.3:0/2308942472 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ea00055c0 con 0x7f1ec80b4390 2023-10-30T12:09:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.615+0000 7f1eaf7fe640 1 -- 172.21.15.3:0/2308942472 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mgrmap(e 3) v1 ==== 44658+0+0 (secure 0 0 0) 0x7f1eb8030a50 con 0x7f1ec80b4390 2023-10-30T12:09:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.615+0000 7f1eaf7fe640 1 --2- 172.21.15.3:0/2308942472 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1eb403a910 0x7f1eb403cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.615+0000 7f1ec659c640 1 --2- 172.21.15.3:0/2308942472 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1eb403a910 0x7f1eb403cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.615+0000 7f1ec659c640 10 monclient: get_auth_request con 0x7f1eb403a910 auth_method 0 2023-10-30T12:09:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.616+0000 7f1eaf7fe640 10 client.4156.objecter ms_handle_connect 0x7f1eb403a910 2023-10-30T12:09:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.616+0000 7f1ec659c640 1 --2- 172.21.15.3:0/2308942472 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1eb403a910 0x7f1eb403cd00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80b0f00 tx=0x7f1ebc007990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:25.716 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:09:25.715+0000 7f679bdff640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-30T12:09:25.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.852+0000 7f1eaf7fe640 1 -- 172.21.15.3:0/2308942472 <== mon.1 v2:172.21.15.92:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f1eb804cd80 con 0x7f1ec80b4390 2023-10-30T12:09:25.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.852+0000 7f1eaf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:25.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.852+0000 7f1eaf7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:25.855 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:09:25.855 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":10,"fsid":"9bae44ae-7379-4ab2-a454-4d550607110c","created":"2023-10-30T12:09:17.680340+0000","modified":"2023-10-30T12:09:24.800567+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:09:24.800567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69005df2-28b6-4bb2-b3e6-79df0d403c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6466ea5c-d74a-4709-a83a-f5d4fe17157e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"69ec5657-9972-4f4b-8012-83886eb47429","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3dad24-4aeb-47a3-ad1d-d13c77cafad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d38d9d7f-b0de-48ab-a850-499c6a0a4364","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b39d2b0a-84ef-4909-bac7-47f146305a79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0a9d81ac-5a14-4fef-877a-982594787dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:09:25.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 10 librados: watch_flush enter 2023-10-30T12:09:25.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 10 librados: watch_flush exit 2023-10-30T12:09:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 20 client.4156.objecter shutdown clearing up homeless session... 2023-10-30T12:09:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 10 client.4156.objecter successfully canceled tick 2023-10-30T12:09:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 1 -- 172.21.15.3:0/2308942472 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1eb403a910 msgr2=0x7f1eb403cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:25.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 1 --2- 172.21.15.3:0/2308942472 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1eb403a910 0x7f1eb403cd00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80b0f00 tx=0x7f1ebc007990 comp rx=0 tx=0).stop 2023-10-30T12:09:25.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 10 monclient: shutdown 2023-10-30T12:09:25.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:25.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 1 -- 172.21.15.3:0/2308942472 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ec80b4390 msgr2=0x7f1ec81da4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:25.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 1 --2- 172.21.15.3:0/2308942472 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ec80b4390 0x7f1ec81da4c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8000c00 tx=0x7f1eb80046c0 comp rx=0 tx=0).stop 2023-10-30T12:09:25.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 1 -- 172.21.15.3:0/2308942472 shutdown_connections 2023-10-30T12:09:25.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 1 --2- 172.21.15.3:0/2308942472 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1eb403a910 0x7f1eb403cd00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:25.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 1 --2- 172.21.15.3:0/2308942472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ec8141ce0 0x7f1ec81357b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:25.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 1 --2- 172.21.15.3:0/2308942472 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1ec81418a0 0x7f1ec8135270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:25.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 1 --2- 172.21.15.3:0/2308942472 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ec80b4390 0x7f1ec81da4c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:25.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 1 -- 172.21.15.3:0/2308942472 >> 172.21.15.3:0/2308942472 conn(0x7f1ec8065ea0 msgr2=0x7f1ec813a860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:25.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 1 -- 172.21.15.3:0/2308942472 shutdown_connections 2023-10-30T12:09:25.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 1 -- 172.21.15.3:0/2308942472 wait complete. 2023-10-30T12:09:25.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:09:25.853+0000 7f1ead7fa640 1 librados: shutdown 2023-10-30T12:09:25.870 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-30T12:09:25.870 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-30T12:09:26.676 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:09:26.828 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.826+0000 7fac6da6f640 10 monclient: get_monmap_and_config 2023-10-30T12:09:26.828 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.826+0000 7fac6da6f640 10 monclient: build_initial_monmap 2023-10-30T12:09:26.828 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.826+0000 7fac6da6f640 10 monclient: monmap: 2023-10-30T12:09:26.829 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2023-10-30T12:09:26.829 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:26.829 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:09:26.827820+0000 2023-10-30T12:09:26.829 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:09:26.827820+0000 2023-10-30T12:09:26.829 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:26.830 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:26.830 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:09:26.830 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:09:26.830 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:09:26.830 INFO:teuthology.misc.health.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:09:26.830 INFO:teuthology.misc.health.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:09:26.831 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:26.833 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.827+0000 7fac6da6f640 1 Processor -- start 2023-10-30T12:09:26.833 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.827+0000 7fac6da6f640 1 -- start start 2023-10-30T12:09:26.833 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.827+0000 7fac6da6f640 10 monclient: init 2023-10-30T12:09:26.833 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.828+0000 7fac6da6f640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:26.833 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.828+0000 7fac6da6f640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-10-30T12:09:26.834 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.828+0000 7fac6da6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac680a8ca0 addr v1:172.21.15.3:6789/0 2023-10-30T12:09:26.834 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.828+0000 7fac6da6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac680a9080 addr v1:172.21.15.92:6789/0 2023-10-30T12:09:26.834 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.828+0000 7fac6da6f640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fac680b1b20 0x7fac6814d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:26.834 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.828+0000 7fac6da6f640 10 monclient(hunting): picked mon.noname-c con 0x7fac680b1b20 addr v2:172.21.15.92:3300/0 2023-10-30T12:09:26.835 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.829+0000 7fac6da6f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac680a9540 con 0x7fac680a8ca0 2023-10-30T12:09:26.835 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.829+0000 7fac6da6f640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac68142620 con 0x7fac680a9080 2023-10-30T12:09:26.835 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.829+0000 7fac6da6f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:26.835 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.829+0000 7fac6da6f640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac68067a30 con 0x7fac680b1b20 2023-10-30T12:09:26.835 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.829+0000 7fac6da6f640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:26.836 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.829+0000 7fac6da6f640 10 monclient(hunting): authenticate will time out at 1358.043099s 2023-10-30T12:09:26.836 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.829+0000 7fac67fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fac680b1b20 0x7fac6814d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:26.836 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.830+0000 7fac677fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fac680a8ca0 0x7fac68141f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55750/0 (socket says 172.21.15.3:55750) 2023-10-30T12:09:26.836 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.830+0000 7fac677fe640 1 -- 172.21.15.3:0/2265389262 learned_addr learned my addr 172.21.15.3:0/2265389262 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:09:26.836 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.830+0000 7fac67fff640 10 monclient(hunting): get_auth_request con 0x7fac680b1b20 auth_method 0 2023-10-30T12:09:26.836 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.830+0000 7fac67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:26.837 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.830+0000 7fac67fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:26.837 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.830+0000 7fac67fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:26.838 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.832+0000 7fac67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:26.838 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.832+0000 7fac67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:26.838 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.832+0000 7fac67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:26.838 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.834+0000 7fac667fc640 1 -- 172.21.15.3:0/2265389262 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413125973 0 0) 0x7fac58001110 con 0x7fac680a8ca0 2023-10-30T12:09:26.839 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.834+0000 7fac667fc640 10 monclient(hunting): _negotiate 2023-10-30T12:09:26.839 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.834+0000 7fac667fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:26.839 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.834+0000 7fac667fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:26.839 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.834+0000 7fac667fc640 10 monclient(hunting): my global_id is 4179 2023-10-30T12:09:26.839 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.834+0000 7fac667fc640 1 -- 172.21.15.3:0/2265389262 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac50003520 con 0x7fac680a8ca0 2023-10-30T12:09:26.840 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.835+0000 7fac667fc640 1 -- 172.21.15.3:0/2265389262 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390075070 0 0) 0x7fac5c001110 con 0x7fac680a9080 2023-10-30T12:09:26.840 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.835+0000 7fac667fc640 10 monclient(hunting): _negotiate 2023-10-30T12:09:26.840 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.835+0000 7fac667fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:26.840 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.835+0000 7fac667fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:26.840 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.835+0000 7fac667fc640 10 monclient(hunting): my global_id is 4162 2023-10-30T12:09:26.840 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.835+0000 7fac667fc640 1 -- 172.21.15.3:0/2265389262 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac50003840 con 0x7fac680a9080 2023-10-30T12:09:26.841 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.837+0000 7fac67fff640 10 monclient(hunting): handle_auth_done global_id 4159 payload 274 2023-10-30T12:09:26.842 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.837+0000 7fac67fff640 1 -- 172.21.15.3:0/2265389262 >> v1:172.21.15.92:6789/0 conn(0x7fac680a9080 legacy=0x7fac6813ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:26.842 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.837+0000 7fac67fff640 1 -- 172.21.15.3:0/2265389262 >> v1:172.21.15.3:6789/0 conn(0x7fac680a8ca0 legacy=0x7fac68141f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:26.842 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.837+0000 7fac67fff640 10 monclient: _finish_hunting 0 2023-10-30T12:09:26.842 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.837+0000 7fac67fff640 1 monclient: found mon.noname-c 2023-10-30T12:09:26.842 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.837+0000 7fac67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:26.843 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.837+0000 7fac67fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:09:26.843 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.837+0000 7fac67fff640 1 -- 172.21.15.3:0/2265389262 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac680b39e0 con 0x7fac680b1b20 2023-10-30T12:09:26.847 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.846+0000 7fac67fff640 1 --2- 172.21.15.3:0/2265389262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac680b1b20 0x7fac6814d170 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fac60009d90 tx=0x7fac6002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef3bc1163b17d9e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:26.851 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.850+0000 7fac667fc640 1 -- 172.21.15.3:0/2265389262 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6002cc60 con 0x7fac680b1b20 2023-10-30T12:09:26.851 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.850+0000 7fac667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:26.851 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.850+0000 7fac667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:26.852 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.850+0000 7fac667fc640 10 monclient: dump: 2023-10-30T12:09:26.852 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:26.852 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:26.852 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:26.852 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:26.853 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:26.853 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:26.853 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:26.853 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:26.853 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:26.854 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:26.854 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.850+0000 7fac667fc640 10 monclient: _finish_auth 0 2023-10-30T12:09:26.854 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.850+0000 7fac667fc640 10 monclient: _check_auth_tickets 2023-10-30T12:09:26.854 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.850+0000 7fac667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:26.855 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.850+0000 7fac6da6f640 5 monclient: authenticate success, global_id 4159 2023-10-30T12:09:26.855 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.850+0000 7fac6da6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:26.861 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.852+0000 7fac667fc640 1 -- 172.21.15.3:0/2265389262 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6002cdd0 con 0x7fac680b1b20 2023-10-30T12:09:26.861 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.852+0000 7fac667fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:26.862 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.852+0000 7fac667fc640 1 -- 172.21.15.3:0/2265389262 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac60034640 con 0x7fac680b1b20 2023-10-30T12:09:26.862 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.852+0000 7fac667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:26.862 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.852+0000 7fac667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:26.862 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.852+0000 7fac667fc640 10 monclient: dump: 2023-10-30T12:09:26.863 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:26.863 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:26.863 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:26.863 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:26.863 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:26.864 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:26.864 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:26.864 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:26.864 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:26.864 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:26.865 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.852+0000 7fac6da6f640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:26.865 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.852+0000 7fac6da6f640 4 set_mon_vals no callback set 2023-10-30T12:09:26.865 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.853+0000 7fac6da6f640 10 monclient: shutdown 2023-10-30T12:09:26.865 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.853+0000 7fac6da6f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:26.866 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.853+0000 7fac6da6f640 1 -- 172.21.15.3:0/2265389262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac680b1b20 msgr2=0x7fac6814d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:26.866 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.853+0000 7fac6da6f640 1 --2- 172.21.15.3:0/2265389262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac680b1b20 0x7fac6814d170 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fac60009d90 tx=0x7fac6002beb0 comp rx=0 tx=0).stop 2023-10-30T12:09:26.866 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.853+0000 7fac6da6f640 1 -- 172.21.15.3:0/2265389262 shutdown_connections 2023-10-30T12:09:26.866 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.853+0000 7fac6da6f640 1 --2- 172.21.15.3:0/2265389262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac680b1b20 0x7fac6814d170 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:26.866 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.853+0000 7fac6da6f640 1 -- 172.21.15.3:0/2265389262 >> 172.21.15.3:0/2265389262 conn(0x7fac680b71a0 msgr2=0x7fac680b75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:26.867 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.853+0000 7fac6da6f640 1 -- 172.21.15.3:0/2265389262 shutdown_connections 2023-10-30T12:09:26.867 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.853+0000 7fac6da6f640 1 -- 172.21.15.3:0/2265389262 wait complete. 2023-10-30T12:09:26.867 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.854+0000 7fac6da6f640 10 monclient: build_initial_monmap 2023-10-30T12:09:26.867 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.854+0000 7fac6da6f640 10 monclient: monmap: 2023-10-30T12:09:26.867 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2023-10-30T12:09:26.867 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:26.868 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2023-10-30T12:09:26.868 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2023-10-30T12:09:26.868 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:26.868 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2023-10-30T12:09:26.868 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:09:26.869 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:09:26.869 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:09:26.869 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:26.869 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.855+0000 7fac6da6f640 1 librados: starting msgr at 2023-10-30T12:09:26.869 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.855+0000 7fac6da6f640 1 librados: starting objecter 2023-10-30T12:09:26.870 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.855+0000 7fac6da6f640 1 Processor -- start 2023-10-30T12:09:26.870 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.855+0000 7fac6da6f640 1 -- start start 2023-10-30T12:09:26.870 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.855+0000 7fac6da6f640 1 librados: setting wanted keys 2023-10-30T12:09:26.870 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.855+0000 7fac6da6f640 1 librados: calling monclient init 2023-10-30T12:09:26.870 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.855+0000 7fac6da6f640 10 monclient: init 2023-10-30T12:09:26.871 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.856+0000 7fac6da6f640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:26.871 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.856+0000 7fac6da6f640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:09:26.871 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.856+0000 7fac6da6f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac680a8ca0 0x7fac680b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:26.871 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.856+0000 7fac6da6f640 10 monclient(hunting): picked mon.noname-a con 0x7fac680a8ca0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:26.871 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.856+0000 7fac6da6f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac680a9080 0x7fac680b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:26.872 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.857+0000 7fac6da6f640 10 monclient(hunting): picked mon.noname-c con 0x7fac680a9080 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:26.872 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.857+0000 7fac6da6f640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac680b1b20 0x7fac680abf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:26.872 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.857+0000 7fac6da6f640 10 monclient(hunting): picked mon.noname-b con 0x7fac680b1b20 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:26.872 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.857+0000 7fac6da6f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:26.872 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.857+0000 7fac6da6f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac680b3270 con 0x7fac680a8ca0 2023-10-30T12:09:26.873 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.857+0000 7fac6da6f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:26.873 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.857+0000 7fac6da6f640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac680b5820 con 0x7fac680b1b20 2023-10-30T12:09:26.873 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.857+0000 7fac677fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac680a8ca0 0x7fac680b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:26.873 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.857+0000 7fac6da6f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:26.873 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.857+0000 7fac6da6f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac680ad8a0 con 0x7fac680a9080 2023-10-30T12:09:26.874 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.857+0000 7fac6da6f640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:26.874 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.857+0000 7fac6da6f640 10 monclient(hunting): authenticate will time out at 1358.071885s 2023-10-30T12:09:26.874 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.857+0000 7fac67fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac680b1b20 0x7fac680abf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:26.874 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac66ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac680a9080 0x7fac680b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:26.874 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac66ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac680a9080 0x7fac680b52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:60828/0 (socket says 172.21.15.3:60828) 2023-10-30T12:09:26.875 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac677fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac680a8ca0 0x7fac680b4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48330/0 (socket says 172.21.15.3:48330) 2023-10-30T12:09:26.875 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac66ffd640 1 -- 172.21.15.3:0/3649579531 learned_addr learned my addr 172.21.15.3:0/3649579531 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:09:26.875 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac66ffd640 10 monclient(hunting): get_auth_request con 0x7fac680a9080 auth_method 0 2023-10-30T12:09:26.875 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:26.875 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac66ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:26.876 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac66ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:26.876 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac677fe640 10 monclient(hunting): get_auth_request con 0x7fac680a8ca0 auth_method 0 2023-10-30T12:09:26.876 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:26.876 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac677fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:26.876 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac677fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:26.877 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac67fff640 10 monclient(hunting): get_auth_request con 0x7fac680b1b20 auth_method 0 2023-10-30T12:09:26.877 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:26.877 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac67fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:26.877 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.858+0000 7fac67fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:26.877 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.859+0000 7fac66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:26.878 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.859+0000 7fac66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:26.878 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.859+0000 7fac66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:26.878 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.859+0000 7fac67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:26.878 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.859+0000 7fac67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:26.878 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.859+0000 7fac67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:26.879 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.859+0000 7fac677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:26.879 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.859+0000 7fac677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:26.879 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.859+0000 7fac677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:26.883 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.863+0000 7fac67fff640 10 monclient(hunting): handle_auth_done global_id 4165 payload 824 2023-10-30T12:09:26.883 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.864+0000 7fac67fff640 1 -- 172.21.15.3:0/3649579531 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac680a9080 msgr2=0x7fac680b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:26.883 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.864+0000 7fac67fff640 1 --2- 172.21.15.3:0/3649579531 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac680a9080 0x7fac680b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:26.884 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.864+0000 7fac67fff640 1 -- 172.21.15.3:0/3649579531 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac680a8ca0 msgr2=0x7fac680b4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:26.884 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.864+0000 7fac67fff640 1 --2- 172.21.15.3:0/3649579531 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac680a8ca0 0x7fac680b4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:26.884 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.864+0000 7fac67fff640 10 monclient: _finish_hunting 0 2023-10-30T12:09:26.884 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.864+0000 7fac67fff640 1 monclient: found mon.noname-b 2023-10-30T12:09:26.884 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.864+0000 7fac67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:26.885 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.864+0000 7fac67fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:09:26.885 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.864+0000 7fac67fff640 1 -- 172.21.15.3:0/3649579531 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac680ab9d0 con 0x7fac680b1b20 2023-10-30T12:09:26.885 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.867+0000 7fac67fff640 1 --2- 172.21.15.3:0/3649579531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac680b1b20 0x7fac680abf00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fac60033000 tx=0x7fac60004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:26.885 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.867+0000 7fac4ffff640 10 client.?.objecter ms_handle_connect 0x7fac680b1b20 2023-10-30T12:09:26.885 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.867+0000 7fac4ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:26.886 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.872+0000 7fac4ffff640 1 -- 172.21.15.3:0/3649579531 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac60004880 con 0x7fac680b1b20 2023-10-30T12:09:26.886 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.872+0000 7fac4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:26.886 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.872+0000 7fac4ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:26.886 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.872+0000 7fac4ffff640 10 monclient: dump: 2023-10-30T12:09:26.887 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:26.887 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:26.887 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:26.887 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:26.887 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:26.887 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:26.888 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:26.888 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:26.888 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:26.888 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:26.889 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.872+0000 7fac4ffff640 10 monclient: _finish_auth 0 2023-10-30T12:09:26.889 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.872+0000 7fac4ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:09:26.889 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.872+0000 7fac4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:26.889 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.872+0000 7fac4ffff640 1 -- 172.21.15.3:0/3649579531 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac60009450 con 0x7fac680b1b20 2023-10-30T12:09:26.889 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.872+0000 7fac4ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:26.890 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.872+0000 7fac4ffff640 1 -- 172.21.15.3:0/3649579531 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac60034640 con 0x7fac680b1b20 2023-10-30T12:09:26.890 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.872+0000 7fac4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:26.890 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.872+0000 7fac4ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:26.890 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.872+0000 7fac4ffff640 10 monclient: dump: 2023-10-30T12:09:26.890 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:26.890 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:26.891 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:26.891 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:26.891 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:26.892 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:26.892 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:26.892 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:26.892 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:26.892 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:26.893 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.872+0000 7fac65ffb640 4 set_mon_vals no callback set 2023-10-30T12:09:26.893 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.873+0000 7fac6da6f640 5 monclient: authenticate success, global_id 4165 2023-10-30T12:09:26.893 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.873+0000 7fac6da6f640 10 monclient: _renew_subs 2023-10-30T12:09:26.893 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.873+0000 7fac6da6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:26.893 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.873+0000 7fac6da6f640 1 -- 172.21.15.3:0/3649579531 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac680b6c30 con 0x7fac680b1b20 2023-10-30T12:09:26.893 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.873+0000 7fac6da6f640 10 client.4165.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:26.894 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.873+0000 7fac6da6f640 10 monclient: _renew_subs 2023-10-30T12:09:26.894 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.873+0000 7fac6da6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:26.894 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.873+0000 7fac6da6f640 1 -- 172.21.15.3:0/3649579531 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac680ac620 con 0x7fac680b1b20 2023-10-30T12:09:26.894 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.873+0000 7fac6da6f640 1 librados: init done 2023-10-30T12:09:26.894 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.874+0000 7fac6da6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:26.895 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.874+0000 7fac6da6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:26.895 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.874+0000 7fac6da6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:26.895 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.874+0000 7fac6da6f640 1 -- 172.21.15.3:0/3649579531 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6804e9b0 con 0x7fac680b1b20 2023-10-30T12:09:26.915 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.913+0000 7fac4ffff640 1 -- 172.21.15.3:0/3649579531 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7fac6003b030 con 0x7fac680b1b20 2023-10-30T12:09:26.916 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.914+0000 7fac4ffff640 1 --2- 172.21.15.3:0/3649579531 >> v2:172.21.15.92:6800/1750155735 conn(0x7fac5406c730 0x7fac5406ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:26.916 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.914+0000 7fac677fe640 1 --2- 172.21.15.3:0/3649579531 >> v2:172.21.15.92:6800/1750155735 conn(0x7fac5406c730 0x7fac5406ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:26.916 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.914+0000 7fac677fe640 10 monclient: get_auth_request con 0x7fac5406c730 auth_method 0 2023-10-30T12:09:26.917 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.915+0000 7fac4ffff640 10 client.4165.objecter ms_handle_connect 0x7fac5406c730 2023-10-30T12:09:26.917 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.915+0000 7fac677fe640 1 --2- 172.21.15.3:0/3649579531 >> v2:172.21.15.92:6800/1750155735 conn(0x7fac5406c730 0x7fac5406ebf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fac58000f60 tx=0x7fac58007bb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:26.920 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.919+0000 7fac4ffff640 1 -- 172.21.15.3:0/3649579531 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fac60003e30 con 0x7fac680b1b20 2023-10-30T12:09:26.921 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.919+0000 7fac4ffff640 10 client.4165.objecter ms_dispatch 0x7fac68000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T12:09:26.921 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.919+0000 7fac4ffff640 3 client.4165.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T12:09:26.921 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.919+0000 7fac4ffff640 3 client.4165.objecter handle_osd_map decoding full epoch 10 2023-10-30T12:09:26.921 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:26.919+0000 7fac4ffff640 20 client.4165.objecter dump_active .. 0 homeless 2023-10-30T12:09:27.351 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.350+0000 7fac4ffff640 1 -- 172.21.15.3:0/3649579531 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fac6007a630 con 0x7fac680b1b20 2023-10-30T12:09:27.352 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.350+0000 7fac4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:27.352 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.350+0000 7fac4ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:27.722 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.720+0000 7fac6da6f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:27.722 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.720+0000 7fac6da6f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:27.723 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.720+0000 7fac6da6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:27.723 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.720+0000 7fac6da6f640 1 -- 172.21.15.3:0/3649579531 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac680ac620 con 0x7fac680b1b20 2023-10-30T12:09:27.751 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.750+0000 7fac4ffff640 1 -- 172.21.15.3:0/3649579531 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fac6007e6d0 con 0x7fac680b1b20 2023-10-30T12:09:27.752 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.750+0000 7fac4ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:27.752 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.750+0000 7fac4ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:27.752 INFO:teuthology.misc.health.smithi003.stdout: 2023-10-30T12:09:27.753 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"9bae44ae-7379-4ab2-a454-4d550607110c","created":"2023-10-30T12:09:17.680340+0000","modified":"2023-10-30T12:09:24.800567+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:09:24.800567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69005df2-28b6-4bb2-b3e6-79df0d403c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6466ea5c-d74a-4709-a83a-f5d4fe17157e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"69ec5657-9972-4f4b-8012-83886eb47429","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3dad24-4aeb-47a3-ad1d-d13c77cafad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d38d9d7f-b0de-48ab-a850-499c6a0a4364","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b39d2b0a-84ef-4909-bac7-47f146305a79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0a9d81ac-5a14-4fef-877a-982594787dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:09:27.756 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.753+0000 7fac6da6f640 10 librados: watch_flush enter 2023-10-30T12:09:27.756 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 10 librados: watch_flush exit 2023-10-30T12:09:27.756 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 20 client.4165.objecter shutdown clearing up homeless session... 2023-10-30T12:09:27.756 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 10 client.4165.objecter successfully canceled tick 2023-10-30T12:09:27.757 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 1 -- 172.21.15.3:0/3649579531 >> v2:172.21.15.92:6800/1750155735 conn(0x7fac5406c730 msgr2=0x7fac5406ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:27.757 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 1 --2- 172.21.15.3:0/3649579531 >> v2:172.21.15.92:6800/1750155735 conn(0x7fac5406c730 0x7fac5406ebf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fac58000f60 tx=0x7fac58007bb0 comp rx=0 tx=0).stop 2023-10-30T12:09:27.757 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 10 monclient: shutdown 2023-10-30T12:09:27.757 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:27.757 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 1 -- 172.21.15.3:0/3649579531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac680b1b20 msgr2=0x7fac680abf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:27.758 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 1 --2- 172.21.15.3:0/3649579531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac680b1b20 0x7fac680abf00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fac60033000 tx=0x7fac60004a40 comp rx=0 tx=0).stop 2023-10-30T12:09:27.758 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 1 -- 172.21.15.3:0/3649579531 shutdown_connections 2023-10-30T12:09:27.758 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 1 --2- 172.21.15.3:0/3649579531 >> v2:172.21.15.92:6800/1750155735 conn(0x7fac5406c730 0x7fac5406ebf0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:27.758 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 1 --2- 172.21.15.3:0/3649579531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac680b1b20 0x7fac680abf00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:27.758 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 1 --2- 172.21.15.3:0/3649579531 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac680a9080 0x7fac680b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:27.759 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 1 --2- 172.21.15.3:0/3649579531 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac680a8ca0 0x7fac680b4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:27.759 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 1 -- 172.21.15.3:0/3649579531 >> 172.21.15.3:0/3649579531 conn(0x7fac680b71a0 msgr2=0x7fac680b2000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:27.759 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 1 -- 172.21.15.3:0/3649579531 shutdown_connections 2023-10-30T12:09:27.759 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 1 -- 172.21.15.3:0/3649579531 wait complete. 2023-10-30T12:09:27.759 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:27.754+0000 7fac6da6f640 1 librados: shutdown 2023-10-30T12:09:27.777 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T12:09:33.017 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:07.979 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:33.021 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:07.981 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:33.047 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:08.004 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:33.081 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:08.042 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:33.088 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:08.049 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:33.120 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:08.077 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:33.372 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:08.323 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:33.431 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:08.382 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:33.636 INFO:tasks.ceph.osd.4.smithi092.stderr:2023-10-30T12:09:33.629+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:09:33.642 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-10-30T12:09:33.636+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:09:33.673 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:08.607 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:33.685 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-10-30T12:09:33.678+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:09:33.691 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:08.644 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:33.720 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:08.676 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:33.734 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:08.668 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:33.760 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:08.713 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:33.780 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:08.737 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:33.802 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:08.734 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:33.866 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:08.798 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:34.005 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-10-30T12:09:33.995+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:09:34.088 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:09.050 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:34.101 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:09.062 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:34.206 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:09.164 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:34.290 INFO:tasks.ceph.osd.2.smithi003.stderr:2023-10-30T12:09:34.280+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:09:34.314 INFO:tasks.ceph.osd.1.smithi003.stderr:2023-10-30T12:09:34.307+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:09:34.342 INFO:tasks.ceph.osd.0.smithi003.stderr:2023-10-30T12:09:34.332+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:09:34.456 INFO:tasks.ceph.osd.3.smithi003.stderr:2023-10-30T12:09:34.448+0000 5dc2340 -1 Falling back to public interface 2023-10-30T12:09:34.465 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:09.427 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:34.484 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:09.445 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:34.488 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:09.439 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:34.583 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:09:34.602 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:09.559 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:34.727 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.725+0000 7f571d3d7640 10 monclient: get_monmap_and_config 2023-10-30T12:09:34.727 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.725+0000 7f571d3d7640 10 monclient: build_initial_monmap 2023-10-30T12:09:34.728 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.725+0000 7f571d3d7640 10 monclient: monmap: 2023-10-30T12:09:34.728 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2023-10-30T12:09:34.728 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:34.728 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:09:34.726688+0000 2023-10-30T12:09:34.729 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:09:34.726688+0000 2023-10-30T12:09:34.729 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:34.729 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:34.729 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:09:34.729 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:09:34.730 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:09:34.730 INFO:teuthology.misc.health.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:09:34.730 INFO:teuthology.misc.health.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:09:34.730 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:34.735 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.726+0000 7f571d3d7640 1 Processor -- start 2023-10-30T12:09:34.735 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.726+0000 7f571d3d7640 1 -- start start 2023-10-30T12:09:34.735 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.726+0000 7f571d3d7640 10 monclient: init 2023-10-30T12:09:34.735 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:34.736 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-10-30T12:09:34.736 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f57180b4350 0x7f57180aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.736 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 10 monclient(hunting): picked mon.noname-c con 0x7f57180b4350 addr v2:172.21.15.92:3300/0 2023-10-30T12:09:34.736 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f57180adc70 0x7f57180b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.737 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 10 monclient(hunting): picked mon.noname-a con 0x7f57180adc70 addr v2:172.21.15.3:3300/0 2023-10-30T12:09:34.737 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f57180b0d50 0x7f571813ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.737 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 10 monclient(hunting): picked mon.noname-b con 0x7f57180b0d50 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:34.737 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.738 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5718067a10 con 0x7f57180adc70 2023-10-30T12:09:34.738 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.738 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57180b4730 con 0x7f57180b4350 2023-10-30T12:09:34.738 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.738 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57180af3b0 con 0x7f57180b0d50 2023-10-30T12:09:34.739 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:34.739 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.727+0000 7f571d3d7640 10 monclient(hunting): authenticate will time out at 1365.941863s 2023-10-30T12:09:34.739 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.728+0000 7f5716ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f57180b4350 0x7f57180aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.739 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.728+0000 7f57177fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f57180b0d50 0x7f571813ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.739 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.728+0000 7f570e7fc640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f57180adc70 0x7f57180b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.740 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.729+0000 7f57177fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f57180b0d50 0x7f571813ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:59616/0 (socket says 172.21.15.3:59616) 2023-10-30T12:09:34.740 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.729+0000 7f57177fe640 1 -- 172.21.15.3:0/2824890051 learned_addr learned my addr 172.21.15.3:0/2824890051 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:09:34.740 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.729+0000 7f57177fe640 10 monclient(hunting): get_auth_request con 0x7f57180b0d50 auth_method 0 2023-10-30T12:09:34.740 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.729+0000 7f57177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.740 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.729+0000 7f57177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.741 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.729+0000 7f57177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.741 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.729+0000 7f570e7fc640 10 monclient(hunting): get_auth_request con 0x7f57180adc70 auth_method 0 2023-10-30T12:09:34.741 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.729+0000 7f570e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.741 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.729+0000 7f570e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.741 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.729+0000 7f570e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.742 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.729+0000 7f5716ffd640 10 monclient(hunting): get_auth_request con 0x7f57180b4350 auth_method 0 2023-10-30T12:09:34.742 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.729+0000 7f5716ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.742 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.730+0000 7f5716ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.742 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.730+0000 7f5716ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.742 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.730+0000 7f57177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.743 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.730+0000 7f57177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.743 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.730+0000 7f57177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.743 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.730+0000 7f570e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.743 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.730+0000 7f570e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.743 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.730+0000 7f570e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.744 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.730+0000 7f5716ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.744 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.730+0000 7f5716ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.744 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.730+0000 7f5716ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.744 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.732+0000 7f5716ffd640 10 monclient(hunting): handle_auth_done global_id 4198 payload 274 2023-10-30T12:09:34.744 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.732+0000 7f5716ffd640 1 -- 172.21.15.3:0/2824890051 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f57180b0d50 msgr2=0x7f571813ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:34.745 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.732+0000 7f5716ffd640 1 --2- 172.21.15.3:0/2824890051 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f57180b0d50 0x7f571813ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.745 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.732+0000 7f5716ffd640 1 -- 172.21.15.3:0/2824890051 >> v2:172.21.15.3:3300/0 conn(0x7f57180adc70 msgr2=0x7f57180b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:34.745 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.732+0000 7f5716ffd640 1 --2- 172.21.15.3:0/2824890051 >> v2:172.21.15.3:3300/0 conn(0x7f57180adc70 0x7f57180b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.745 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.732+0000 7f5716ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:09:34.745 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.732+0000 7f5716ffd640 1 monclient: found mon.noname-c 2023-10-30T12:09:34.746 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.732+0000 7f5716ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:34.746 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.732+0000 7f5716ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:09:34.746 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.732+0000 7f5716ffd640 1 -- 172.21.15.3:0/2824890051 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57180ae950 con 0x7f57180b4350 2023-10-30T12:09:34.756 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.734+0000 7f5716ffd640 1 --2- 172.21.15.3:0/2824890051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57180b4350 0x7f57180aee70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f57180b3340 tx=0x7f570802b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f197abab72b42c06 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:34.756 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.737+0000 7f57167fc640 1 -- 172.21.15.3:0/2824890051 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570802f050 con 0x7f57180b4350 2023-10-30T12:09:34.757 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f57167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:34.757 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f57167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:34.757 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f57167fc640 10 monclient: dump: 2023-10-30T12:09:34.757 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:34.758 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:34.758 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:34.758 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:34.758 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:34.758 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:34.759 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:34.759 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:34.759 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:34.759 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:34.759 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f57167fc640 10 monclient: _finish_auth 0 2023-10-30T12:09:34.759 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f57167fc640 10 monclient: _check_auth_tickets 2023-10-30T12:09:34.760 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f57167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:34.760 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f57167fc640 1 -- 172.21.15.3:0/2824890051 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5708031710 con 0x7f57180b4350 2023-10-30T12:09:34.760 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f571d3d7640 5 monclient: authenticate success, global_id 4198 2023-10-30T12:09:34.760 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f571d3d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:34.760 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f57167fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:34.761 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f57167fc640 1 -- 172.21.15.3:0/2824890051 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5708031ab0 con 0x7f57180b4350 2023-10-30T12:09:34.761 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f571d3d7640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:34.761 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f571d3d7640 4 set_mon_vals no callback set 2023-10-30T12:09:34.761 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f57167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:34.761 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f57167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:34.761 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f57167fc640 10 monclient: dump: 2023-10-30T12:09:34.762 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:34.762 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:34.762 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:34.762 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:34.762 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:34.763 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:34.763 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:34.763 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:34.763 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:34.763 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:34.763 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f571d3d7640 10 monclient: shutdown 2023-10-30T12:09:34.764 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f571d3d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:34.764 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f571d3d7640 1 -- 172.21.15.3:0/2824890051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57180b4350 msgr2=0x7f57180aee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.764 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.738+0000 7f571d3d7640 1 --2- 172.21.15.3:0/2824890051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57180b4350 0x7f57180aee70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f57180b3340 tx=0x7f570802b7b0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.764 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.739+0000 7f571d3d7640 1 -- 172.21.15.3:0/2824890051 shutdown_connections 2023-10-30T12:09:34.764 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.739+0000 7f571d3d7640 1 --2- 172.21.15.3:0/2824890051 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f57180b0d50 0x7f571813ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.765 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.739+0000 7f571d3d7640 1 --2- 172.21.15.3:0/2824890051 >> v2:172.21.15.3:3300/0 conn(0x7f57180adc70 0x7f57180b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.765 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.739+0000 7f571d3d7640 1 --2- 172.21.15.3:0/2824890051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57180b4350 0x7f57180aee70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.765 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.739+0000 7f571d3d7640 1 -- 172.21.15.3:0/2824890051 >> 172.21.15.3:0/2824890051 conn(0x7f5718060bf0 msgr2=0x7f5718061000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:34.765 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.739+0000 7f571d3d7640 1 -- 172.21.15.3:0/2824890051 shutdown_connections 2023-10-30T12:09:34.765 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.739+0000 7f571d3d7640 1 -- 172.21.15.3:0/2824890051 wait complete. 2023-10-30T12:09:34.765 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.740+0000 7f571d3d7640 10 monclient: build_initial_monmap 2023-10-30T12:09:34.766 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.740+0000 7f571d3d7640 10 monclient: monmap: 2023-10-30T12:09:34.766 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2023-10-30T12:09:34.766 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:34.766 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2023-10-30T12:09:34.767 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2023-10-30T12:09:34.767 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:34.767 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2023-10-30T12:09:34.767 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:09:34.767 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:09:34.768 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:09:34.768 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:34.768 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.740+0000 7f571d3d7640 1 librados: starting msgr at 2023-10-30T12:09:34.768 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.740+0000 7f571d3d7640 1 librados: starting objecter 2023-10-30T12:09:34.768 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.740+0000 7f571d3d7640 1 Processor -- start 2023-10-30T12:09:34.768 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 1 -- start start 2023-10-30T12:09:34.769 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 1 librados: setting wanted keys 2023-10-30T12:09:34.769 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 1 librados: calling monclient init 2023-10-30T12:09:34.769 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 10 monclient: init 2023-10-30T12:09:34.769 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:34.769 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:09:34.770 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f57180adc70 0x7f5718070e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.770 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 10 monclient(hunting): picked mon.noname-c con 0x7f57180adc70 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:34.770 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57180b0d50 0x7f57181daeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.770 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 10 monclient(hunting): picked mon.noname-a con 0x7f57180b0d50 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:34.770 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57180b4350 0x7f57180ac990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.770 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 10 monclient(hunting): picked mon.noname-b con 0x7f57180b4350 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:34.771 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.771 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57180b1310 con 0x7f57180b0d50 2023-10-30T12:09:34.771 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.771 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57180a9240 con 0x7f57180b4350 2023-10-30T12:09:34.771 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.772 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57181da990 con 0x7f57180adc70 2023-10-30T12:09:34.772 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:34.772 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.741+0000 7f571d3d7640 10 monclient(hunting): authenticate will time out at 1365.956035s 2023-10-30T12:09:34.772 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.742+0000 7f57177fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57180b4350 0x7f57180ac990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.772 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.742+0000 7f5716ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f57180adc70 0x7f5718070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.773 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.742+0000 7f57177fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57180b4350 0x7f57180ac990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:58326/0 (socket says 172.21.15.3:58326) 2023-10-30T12:09:34.773 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f57177fe640 1 -- 172.21.15.3:0/2424460843 learned_addr learned my addr 172.21.15.3:0/2424460843 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:09:34.773 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f57177fe640 10 monclient(hunting): get_auth_request con 0x7f57180b4350 auth_method 0 2023-10-30T12:09:34.773 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f57177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.773 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f57177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.774 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f57177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.774 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f5716ffd640 10 monclient(hunting): get_auth_request con 0x7f57180adc70 auth_method 0 2023-10-30T12:09:34.774 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f5716ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.774 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f5716ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.774 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f5716ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.775 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f570e7fc640 1 --2- 172.21.15.3:0/2424460843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57180b0d50 0x7f57181daeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.775 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f57177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.775 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f57177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.775 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f57177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.775 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f570e7fc640 10 monclient(hunting): get_auth_request con 0x7f57180b0d50 auth_method 0 2023-10-30T12:09:34.776 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f570e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.776 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f570e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.776 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.743+0000 7f570e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.776 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.744+0000 7f5716ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.776 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.744+0000 7f5716ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.776 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.744+0000 7f5716ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.777 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.744+0000 7f570e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.777 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.744+0000 7f570e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.777 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.744+0000 7f570e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.777 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.746+0000 7f57177fe640 10 monclient(hunting): handle_auth_done global_id 4201 payload 824 2023-10-30T12:09:34.777 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.746+0000 7f57177fe640 1 -- 172.21.15.3:0/2424460843 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f57180adc70 msgr2=0x7f5718070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.778 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.746+0000 7f57177fe640 1 --2- 172.21.15.3:0/2424460843 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f57180adc70 0x7f5718070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.778 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.746+0000 7f57177fe640 1 -- 172.21.15.3:0/2424460843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57180b0d50 msgr2=0x7f57181daeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.778 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.746+0000 7f57177fe640 1 --2- 172.21.15.3:0/2424460843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57180b0d50 0x7f57181daeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.778 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.746+0000 7f57177fe640 10 monclient: _finish_hunting 0 2023-10-30T12:09:34.778 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.746+0000 7f57177fe640 1 monclient: found mon.noname-b 2023-10-30T12:09:34.779 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.746+0000 7f57177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:34.779 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.746+0000 7f57177fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:09:34.779 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.746+0000 7f57177fe640 1 -- 172.21.15.3:0/2424460843 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57180a83d0 con 0x7f57180b4350 2023-10-30T12:09:34.779 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.748+0000 7f57177fe640 1 --2- 172.21.15.3:0/2424460843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57180b4350 0x7f57180ac990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f571000efc0 tx=0x7f571000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:34.779 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.748+0000 7f570ffff640 10 client.?.objecter ms_handle_connect 0x7f57180b4350 2023-10-30T12:09:34.780 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.748+0000 7f570ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:34.780 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.751+0000 7f570ffff640 1 -- 172.21.15.3:0/2424460843 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571000f040 con 0x7f57180b4350 2023-10-30T12:09:34.780 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.751+0000 7f570ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:34.780 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.751+0000 7f570ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:34.780 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.751+0000 7f570ffff640 10 monclient: dump: 2023-10-30T12:09:34.780 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:34.781 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:34.781 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:34.781 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:34.781 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:34.781 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:34.781 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:34.782 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:34.782 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:34.782 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:34.782 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.751+0000 7f570ffff640 10 monclient: _finish_auth 0 2023-10-30T12:09:34.782 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.751+0000 7f570ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:09:34.783 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.751+0000 7f570ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:34.783 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.751+0000 7f570ffff640 1 -- 172.21.15.3:0/2424460843 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571000cd40 con 0x7f57180b4350 2023-10-30T12:09:34.783 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.751+0000 7f570ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:34.783 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.751+0000 7f570ffff640 1 -- 172.21.15.3:0/2424460843 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710003bd0 con 0x7f57180b4350 2023-10-30T12:09:34.783 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.751+0000 7f570ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:34.784 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.751+0000 7f570ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:34.784 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.751+0000 7f570ffff640 10 monclient: dump: 2023-10-30T12:09:34.784 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:34.784 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:34.784 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:34.785 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:34.785 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:34.785 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:34.785 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:34.785 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:34.785 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:34.786 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:34.786 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.751+0000 7f5715ffb640 4 set_mon_vals no callback set 2023-10-30T12:09:34.786 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.752+0000 7f571d3d7640 5 monclient: authenticate success, global_id 4201 2023-10-30T12:09:34.786 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.752+0000 7f571d3d7640 10 monclient: _renew_subs 2023-10-30T12:09:34.786 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.752+0000 7f571d3d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:34.787 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.752+0000 7f571d3d7640 1 -- 172.21.15.3:0/2424460843 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571806ff70 con 0x7f57180b4350 2023-10-30T12:09:34.787 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.752+0000 7f571d3d7640 10 client.4201.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:34.787 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.752+0000 7f571d3d7640 10 monclient: _renew_subs 2023-10-30T12:09:34.787 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.752+0000 7f571d3d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:34.787 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.752+0000 7f571d3d7640 1 -- 172.21.15.3:0/2424460843 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571806e500 con 0x7f57180b4350 2023-10-30T12:09:34.787 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.752+0000 7f571d3d7640 1 librados: init done 2023-10-30T12:09:34.788 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.753+0000 7f571d3d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:34.788 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.753+0000 7f571d3d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:34.788 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.753+0000 7f571d3d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:34.788 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:34.753+0000 7f571d3d7640 1 -- 172.21.15.3:0/2424460843 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5718069650 con 0x7f57180b4350 2023-10-30T12:09:34.791 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:09.725 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:34.809 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:09.771 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:34.812 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:09.765 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:34.815 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:09.771 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:34.846 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:09.807 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:34.868 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:09.819 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:34.942 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:09.899 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.012 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:09.944 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.050 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.048+0000 7f570ffff640 1 -- 172.21.15.3:0/2424460843 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7f5710003d40 con 0x7f57180b4350 2023-10-30T12:09:35.050 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.049+0000 7f570ffff640 1 --2- 172.21.15.3:0/2424460843 >> v2:172.21.15.92:6800/1750155735 conn(0x7f56f806c780 0x7f56f806ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:35.051 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.049+0000 7f5716ffd640 1 --2- 172.21.15.3:0/2424460843 >> v2:172.21.15.92:6800/1750155735 conn(0x7f56f806c780 0x7f56f806ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:35.051 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.049+0000 7f5716ffd640 10 monclient: get_auth_request con 0x7f56f806c780 auth_method 0 2023-10-30T12:09:35.051 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.050+0000 7f5716ffd640 1 --2- 172.21.15.3:0/2424460843 >> v2:172.21.15.92:6800/1750155735 conn(0x7f56f806c780 0x7f56f806ec40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5708035000 tx=0x7f5708008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:35.052 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.050+0000 7f570ffff640 10 client.4201.objecter ms_handle_connect 0x7f56f806c780 2023-10-30T12:09:35.062 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.062+0000 7f570ffff640 1 -- 172.21.15.3:0/2424460843 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f57100591e0 con 0x7f57180b4350 2023-10-30T12:09:35.063 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.062+0000 7f570ffff640 10 client.4201.objecter ms_dispatch 0x7f5718000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T12:09:35.063 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.062+0000 7f570ffff640 3 client.4201.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T12:09:35.063 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.062+0000 7f570ffff640 3 client.4201.objecter handle_osd_map decoding full epoch 10 2023-10-30T12:09:35.063 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.062+0000 7f570ffff640 20 client.4201.objecter dump_active .. 0 homeless 2023-10-30T12:09:35.064 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.062+0000 7f570ffff640 1 -- 172.21.15.3:0/2424460843 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f571005d280 con 0x7f57180b4350 2023-10-30T12:09:35.064 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.062+0000 7f570ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:35.064 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.062+0000 7f570ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:35.097 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:10.059 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.167 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:10.118 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.168 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:10.129 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.198 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:10.131 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.214 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:10.168 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.219 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:10.176 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.238 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:10.196 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.267 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.266+0000 7f571d3d7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:35.268 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.266+0000 7f571d3d7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:35.268 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.266+0000 7f571d3d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:35.268 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.266+0000 7f571d3d7640 1 -- 172.21.15.3:0/2424460843 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f571806e500 con 0x7f57180b4350 2023-10-30T12:09:35.282 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.281+0000 7f570ffff640 1 -- 172.21.15.3:0/2424460843 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f5710061430 con 0x7f57180b4350 2023-10-30T12:09:35.282 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.281+0000 7f570ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:35.282 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.281+0000 7f570ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:35.283 INFO:teuthology.misc.health.smithi003.stdout: 2023-10-30T12:09:35.283 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"9bae44ae-7379-4ab2-a454-4d550607110c","created":"2023-10-30T12:09:17.680340+0000","modified":"2023-10-30T12:09:24.800567+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:09:24.800567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69005df2-28b6-4bb2-b3e6-79df0d403c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6466ea5c-d74a-4709-a83a-f5d4fe17157e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"69ec5657-9972-4f4b-8012-83886eb47429","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3dad24-4aeb-47a3-ad1d-d13c77cafad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d38d9d7f-b0de-48ab-a850-499c6a0a4364","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b39d2b0a-84ef-4909-bac7-47f146305a79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0a9d81ac-5a14-4fef-877a-982594787dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:09:35.284 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 10 librados: watch_flush enter 2023-10-30T12:09:35.284 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 10 librados: watch_flush exit 2023-10-30T12:09:35.285 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 20 client.4201.objecter shutdown clearing up homeless session... 2023-10-30T12:09:35.285 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 10 client.4201.objecter successfully canceled tick 2023-10-30T12:09:35.285 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 1 -- 172.21.15.3:0/2424460843 >> v2:172.21.15.92:6800/1750155735 conn(0x7f56f806c780 msgr2=0x7f56f806ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:35.285 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 1 --2- 172.21.15.3:0/2424460843 >> v2:172.21.15.92:6800/1750155735 conn(0x7f56f806c780 0x7f56f806ec40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5708035000 tx=0x7f5708008030 comp rx=0 tx=0).stop 2023-10-30T12:09:35.285 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 10 monclient: shutdown 2023-10-30T12:09:35.286 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:35.286 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 1 -- 172.21.15.3:0/2424460843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57180b4350 msgr2=0x7f57180ac990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:35.286 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 1 --2- 172.21.15.3:0/2424460843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57180b4350 0x7f57180ac990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f571000efc0 tx=0x7f571000c490 comp rx=0 tx=0).stop 2023-10-30T12:09:35.286 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 1 -- 172.21.15.3:0/2424460843 shutdown_connections 2023-10-30T12:09:35.287 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 1 --2- 172.21.15.3:0/2424460843 >> v2:172.21.15.92:6800/1750155735 conn(0x7f56f806c780 0x7f56f806ec40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:35.287 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 1 --2- 172.21.15.3:0/2424460843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57180b4350 0x7f57180ac990 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:35.287 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 1 --2- 172.21.15.3:0/2424460843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57180b0d50 0x7f57181daeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:35.287 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 1 --2- 172.21.15.3:0/2424460843 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f57180adc70 0x7f5718070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:35.287 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.282+0000 7f571d3d7640 1 -- 172.21.15.3:0/2424460843 >> 172.21.15.3:0/2424460843 conn(0x7f5718060bf0 msgr2=0x7f57180b2280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:35.288 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.283+0000 7f571d3d7640 1 -- 172.21.15.3:0/2424460843 shutdown_connections 2023-10-30T12:09:35.288 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.283+0000 7f571d3d7640 1 -- 172.21.15.3:0/2424460843 wait complete. 2023-10-30T12:09:35.288 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:35.283+0000 7f571d3d7640 1 librados: shutdown 2023-10-30T12:09:35.293 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T12:09:35.365 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:10.327 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.399 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:10.331 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.439 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:10.390 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.448 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:10.409 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.491 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:10.425 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.505 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:10.463 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.514 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:10.467 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.522 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:10.478 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.630 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:10.591 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.722 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:10.631 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.722 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:10.666 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.722 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:10.679 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.756 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:10.689 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.774 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:10.731 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.787 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:10.740 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.793 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:10.750 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.895 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:10.856 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.975 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:10.907 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.981 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:10.933 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:35.991 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:10.951 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.023 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:10.957 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.038 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:10.995 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.052 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:11.005 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.076 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:11.032 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.162 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:11.124 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.239 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:11.171 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.249 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:11.201 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.269 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:11.230 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.287 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:11.221 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.303 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:11.261 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.320 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:11.274 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.351 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:11.307 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.429 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:11.390 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.507 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:11.439 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.513 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:11.465 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.546 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:11.507 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.552 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:11.486 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.573 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:11.531 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.589 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:11.542 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.617 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:11.573 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.772 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:11.704 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.790 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:11.742 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.818 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:11.751 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.857 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:11.809 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.889 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:11.845 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.917 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:11.878 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:36.948 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:11.909 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.041 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:12.002 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.057 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:11.989 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.072 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:12.033 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.099 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:12.032 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.115 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:12.072 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.136 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:12.089 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.150 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:12.108 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.174 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:12.131 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.343 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:12.294 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.348 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:12.280 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.374 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:12.325 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.657 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:12.591 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.707 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:12.660 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.709 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:12.643 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.763 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:12.719 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.764 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:12.718 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.808 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:12.765 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.941 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:12.873 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:37.986 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:12.918 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:42.100 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:09:42.207 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.205+0000 7fc188a4b640 10 monclient: get_monmap_and_config 2023-10-30T12:09:42.207 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.205+0000 7fc188a4b640 10 monclient: build_initial_monmap 2023-10-30T12:09:42.207 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.205+0000 7fc188a4b640 10 monclient: monmap: 2023-10-30T12:09:42.208 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2023-10-30T12:09:42.208 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:42.208 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:09:42.206428+0000 2023-10-30T12:09:42.208 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:09:42.206428+0000 2023-10-30T12:09:42.209 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:42.209 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:42.209 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:09:42.209 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:09:42.209 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:09:42.210 INFO:teuthology.misc.health.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:09:42.210 INFO:teuthology.misc.health.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:09:42.210 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:42.210 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.206+0000 7fc188a4b640 1 Processor -- start 2023-10-30T12:09:42.211 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.206+0000 7fc188a4b640 1 -- start start 2023-10-30T12:09:42.211 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.206+0000 7fc188a4b640 10 monclient: init 2023-10-30T12:09:42.214 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.206+0000 7fc188a4b640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:42.215 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.206+0000 7fc188a4b640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-10-30T12:09:42.215 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.206+0000 7fc188a4b640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc1840b4350 0x7fc1840aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:42.215 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.206+0000 7fc188a4b640 10 monclient(hunting): picked mon.noname-a con 0x7fc1840b4350 addr v2:172.21.15.3:3300/0 2023-10-30T12:09:42.215 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.206+0000 7fc188a4b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1840adc70 0x7fc1840b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:42.216 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.207+0000 7fc188a4b640 10 monclient(hunting): picked mon.noname-b con 0x7fc1840adc70 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:42.216 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.207+0000 7fc188a4b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc1840b0d50 0x7fc18413ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:42.216 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.207+0000 7fc188a4b640 10 monclient(hunting): picked mon.noname-c con 0x7fc1840b0d50 addr v2:172.21.15.92:3300/0 2023-10-30T12:09:42.216 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.207+0000 7fc188a4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:42.216 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.207+0000 7fc188a4b640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc184067a10 con 0x7fc1840b4350 2023-10-30T12:09:42.217 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.207+0000 7fc188a4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:42.217 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.207+0000 7fc188a4b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1840b4730 con 0x7fc1840b0d50 2023-10-30T12:09:42.217 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.207+0000 7fc188a4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:42.217 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.207+0000 7fc188a4b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1840af3b0 con 0x7fc1840adc70 2023-10-30T12:09:42.217 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.207+0000 7fc188a4b640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:42.218 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.207+0000 7fc188a4b640 10 monclient(hunting): authenticate will time out at 1373.421227s 2023-10-30T12:09:42.218 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.207+0000 7fc18259c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc1840b4350 0x7fc1840aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:42.218 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.207+0000 7fc181d9b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1840adc70 0x7fc1840b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:42.218 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc18259c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc1840b4350 0x7fc1840aee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49892/0 (socket says 172.21.15.3:49892) 2023-10-30T12:09:42.219 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc182d9d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc1840b0d50 0x7fc18413ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:42.219 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc18259c640 1 -- 172.21.15.3:0/642985986 learned_addr learned my addr 172.21.15.3:0/642985986 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:09:42.219 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc18259c640 10 monclient(hunting): get_auth_request con 0x7fc1840b4350 auth_method 0 2023-10-30T12:09:42.219 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc18259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:42.219 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc18259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:42.220 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc18259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:42.220 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc181d9b640 10 monclient(hunting): get_auth_request con 0x7fc1840adc70 auth_method 0 2023-10-30T12:09:42.220 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc181d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:42.220 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc181d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:42.220 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc181d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:42.221 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc181d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:42.221 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc181d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:42.221 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc181d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:42.221 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc18259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:42.221 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc18259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:42.222 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.208+0000 7fc18259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:42.222 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.210+0000 7fc181d9b640 10 monclient(hunting): handle_auth_done global_id 4160 payload 274 2023-10-30T12:09:42.222 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.210+0000 7fc181d9b640 1 -- 172.21.15.3:0/642985986 >> v2:172.21.15.92:3300/0 conn(0x7fc1840b0d50 msgr2=0x7fc18413ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:42.222 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.210+0000 7fc181d9b640 1 --2- 172.21.15.3:0/642985986 >> v2:172.21.15.92:3300/0 conn(0x7fc1840b0d50 0x7fc18413ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:42.222 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.210+0000 7fc181d9b640 1 -- 172.21.15.3:0/642985986 >> v2:172.21.15.3:3300/0 conn(0x7fc1840b4350 msgr2=0x7fc1840aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:42.223 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.210+0000 7fc181d9b640 1 --2- 172.21.15.3:0/642985986 >> v2:172.21.15.3:3300/0 conn(0x7fc1840b4350 0x7fc1840aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:42.223 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.210+0000 7fc181d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:09:42.223 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.210+0000 7fc181d9b640 1 monclient: found mon.noname-b 2023-10-30T12:09:42.223 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.210+0000 7fc181d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:42.223 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.210+0000 7fc181d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:09:42.224 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.210+0000 7fc181d9b640 1 -- 172.21.15.3:0/642985986 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1840ae950 con 0x7fc1840adc70 2023-10-30T12:09:42.224 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.213+0000 7fc181d9b640 1 --2- 172.21.15.3:0/642985986 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1840adc70 0x7fc1840b0810 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc1840b1720 tx=0x7fc17802b7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=804c006d7a3235a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:42.230 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc18159a640 1 -- 172.21.15.3:0/642985986 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc178033620 con 0x7fc1840adc70 2023-10-30T12:09:42.231 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc18159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:42.231 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc18159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:42.231 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc18159a640 10 monclient: dump: 2023-10-30T12:09:42.231 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:42.231 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:42.232 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:42.232 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:42.232 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:42.232 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:42.232 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:42.233 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:42.233 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:42.233 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:42.233 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc18159a640 10 monclient: _finish_auth 0 2023-10-30T12:09:42.233 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc18159a640 10 monclient: _check_auth_tickets 2023-10-30T12:09:42.234 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc18159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:42.234 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc18159a640 1 -- 172.21.15.3:0/642985986 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc178033790 con 0x7fc1840adc70 2023-10-30T12:09:42.234 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc18159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:42.234 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc18159a640 1 -- 172.21.15.3:0/642985986 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc178033b70 con 0x7fc1840adc70 2023-10-30T12:09:42.235 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc18159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:42.235 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc18159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:42.235 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc18159a640 10 monclient: dump: 2023-10-30T12:09:42.235 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:42.235 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:42.236 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:42.236 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:42.236 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:42.236 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:42.236 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:42.237 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:42.237 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:42.237 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:42.237 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc188a4b640 5 monclient: authenticate success, global_id 4160 2023-10-30T12:09:42.237 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc188a4b640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:42.238 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.217+0000 7fc188a4b640 4 set_mon_vals no callback set 2023-10-30T12:09:42.238 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.218+0000 7fc188a4b640 10 monclient: shutdown 2023-10-30T12:09:42.238 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.218+0000 7fc188a4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:42.238 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.218+0000 7fc188a4b640 1 -- 172.21.15.3:0/642985986 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1840adc70 msgr2=0x7fc1840b0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:42.238 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.218+0000 7fc188a4b640 1 --2- 172.21.15.3:0/642985986 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1840adc70 0x7fc1840b0810 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc1840b1720 tx=0x7fc17802b7d0 comp rx=0 tx=0).stop 2023-10-30T12:09:42.239 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.219+0000 7fc188a4b640 1 -- 172.21.15.3:0/642985986 shutdown_connections 2023-10-30T12:09:42.239 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.219+0000 7fc188a4b640 1 --2- 172.21.15.3:0/642985986 >> v2:172.21.15.92:3300/0 conn(0x7fc1840b0d50 0x7fc18413ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:42.239 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.219+0000 7fc188a4b640 1 --2- 172.21.15.3:0/642985986 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1840adc70 0x7fc1840b0810 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:42.239 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.219+0000 7fc188a4b640 1 --2- 172.21.15.3:0/642985986 >> v2:172.21.15.3:3300/0 conn(0x7fc1840b4350 0x7fc1840aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:42.240 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.219+0000 7fc188a4b640 1 -- 172.21.15.3:0/642985986 >> 172.21.15.3:0/642985986 conn(0x7fc184060bf0 msgr2=0x7fc184061000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:42.240 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.219+0000 7fc188a4b640 1 -- 172.21.15.3:0/642985986 shutdown_connections 2023-10-30T12:09:42.240 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.219+0000 7fc188a4b640 1 -- 172.21.15.3:0/642985986 wait complete. 2023-10-30T12:09:42.240 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.220+0000 7fc188a4b640 10 monclient: build_initial_monmap 2023-10-30T12:09:42.240 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.220+0000 7fc188a4b640 10 monclient: monmap: 2023-10-30T12:09:42.241 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2023-10-30T12:09:42.241 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:42.241 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2023-10-30T12:09:42.241 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2023-10-30T12:09:42.241 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:42.242 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2023-10-30T12:09:42.242 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:09:42.242 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:09:42.242 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:09:42.242 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:42.243 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.220+0000 7fc188a4b640 1 librados: starting msgr at 2023-10-30T12:09:42.243 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.220+0000 7fc188a4b640 1 librados: starting objecter 2023-10-30T12:09:42.243 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.220+0000 7fc188a4b640 1 Processor -- start 2023-10-30T12:09:42.243 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.220+0000 7fc188a4b640 1 -- start start 2023-10-30T12:09:42.243 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.220+0000 7fc188a4b640 1 librados: setting wanted keys 2023-10-30T12:09:42.243 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.220+0000 7fc188a4b640 1 librados: calling monclient init 2023-10-30T12:09:42.244 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.220+0000 7fc188a4b640 10 monclient: init 2023-10-30T12:09:42.244 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:42.244 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:09:42.244 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1840adc70 0x7fc1840accf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:42.244 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 10 monclient(hunting): picked mon.noname-c con 0x7fc1840adc70 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:42.245 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1840b0d50 0x7fc1840aa010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:42.245 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 10 monclient(hunting): picked mon.noname-a con 0x7fc1840b0d50 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:42.245 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1840b4350 0x7fc18411f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:42.245 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 10 monclient(hunting): picked mon.noname-b con 0x7fc1840b4350 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:42.245 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:42.246 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1840ae5b0 con 0x7fc1840b0d50 2023-10-30T12:09:42.246 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:42.246 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1841d6740 con 0x7fc1840b4350 2023-10-30T12:09:42.246 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:42.246 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1841d6a20 con 0x7fc1840adc70 2023-10-30T12:09:42.247 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:42.247 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc188a4b640 10 monclient(hunting): authenticate will time out at 1373.435431s 2023-10-30T12:09:42.247 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc181d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1840b0d50 0x7fc1840aa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:42.247 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.221+0000 7fc18259c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1840adc70 0x7fc1840accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:42.247 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.222+0000 7fc182d9d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1840b4350 0x7fc18411f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:42.248 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.222+0000 7fc181d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1840b0d50 0x7fc1840aa010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49904/0 (socket says 172.21.15.3:49904) 2023-10-30T12:09:42.248 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.222+0000 7fc18259c640 10 monclient(hunting): get_auth_request con 0x7fc1840adc70 auth_method 0 2023-10-30T12:09:42.248 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.222+0000 7fc181d9b640 1 -- 172.21.15.3:0/2704447050 learned_addr learned my addr 172.21.15.3:0/2704447050 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:09:42.248 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.222+0000 7fc18259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:42.248 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.222+0000 7fc18259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:42.248 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.222+0000 7fc18259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:42.249 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.222+0000 7fc181d9b640 10 monclient(hunting): get_auth_request con 0x7fc1840b0d50 auth_method 0 2023-10-30T12:09:42.249 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.222+0000 7fc181d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:42.249 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.222+0000 7fc181d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:42.249 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.222+0000 7fc181d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:42.249 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.222+0000 7fc18259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:42.250 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.222+0000 7fc18259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:42.250 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.222+0000 7fc18259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:42.250 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.223+0000 7fc181d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:42.250 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.223+0000 7fc181d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:42.250 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.223+0000 7fc181d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:42.251 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.223+0000 7fc182d9d640 10 monclient(hunting): get_auth_request con 0x7fc1840b4350 auth_method 0 2023-10-30T12:09:42.251 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.223+0000 7fc182d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:42.251 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.223+0000 7fc182d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:42.251 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.223+0000 7fc182d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:42.251 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.223+0000 7fc182d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:42.251 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.223+0000 7fc182d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:42.252 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.223+0000 7fc182d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:42.252 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.225+0000 7fc18259c640 10 monclient(hunting): handle_auth_done global_id 4163 payload 824 2023-10-30T12:09:42.252 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.225+0000 7fc18259c640 1 -- 172.21.15.3:0/2704447050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1840b4350 msgr2=0x7fc18411f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:42.252 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.225+0000 7fc18259c640 1 --2- 172.21.15.3:0/2704447050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1840b4350 0x7fc18411f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:42.252 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.225+0000 7fc18259c640 1 -- 172.21.15.3:0/2704447050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1840b0d50 msgr2=0x7fc1840aa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:42.252 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.225+0000 7fc18259c640 1 --2- 172.21.15.3:0/2704447050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1840b0d50 0x7fc1840aa010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:42.253 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.225+0000 7fc18259c640 10 monclient: _finish_hunting 0 2023-10-30T12:09:42.253 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.225+0000 7fc18259c640 1 monclient: found mon.noname-c 2023-10-30T12:09:42.253 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.225+0000 7fc18259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:42.253 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.225+0000 7fc18259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.3:3301/0 2023-10-30T12:09:42.253 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.225+0000 7fc18259c640 1 -- 172.21.15.3:0/2704447050 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc184120760 con 0x7fc1840adc70 2023-10-30T12:09:42.254 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.228+0000 7fc18259c640 1 --2- 172.21.15.3:0/2704447050 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1840adc70 0x7fc1840accf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc17400e870 tx=0x7fc17400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:42.254 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.228+0000 7fc172ffd640 10 client.?.objecter ms_handle_connect 0x7fc1840adc70 2023-10-30T12:09:42.254 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.228+0000 7fc172ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:42.258 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.232+0000 7fc172ffd640 1 -- 172.21.15.3:0/2704447050 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17400ee60 con 0x7fc1840adc70 2023-10-30T12:09:42.259 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.232+0000 7fc172ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:42.259 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.232+0000 7fc172ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:42.259 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.232+0000 7fc172ffd640 10 monclient: dump: 2023-10-30T12:09:42.259 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:42.259 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:42.260 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:42.260 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:42.260 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:42.260 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:42.261 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:42.261 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:42.261 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:42.261 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:42.261 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc172ffd640 10 monclient: _finish_auth 0 2023-10-30T12:09:42.262 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc172ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:09:42.262 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc172ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:42.262 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc172ffd640 1 -- 172.21.15.3:0/2704447050 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17400d040 con 0x7fc1840adc70 2023-10-30T12:09:42.262 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc172ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:42.262 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc172ffd640 1 -- 172.21.15.3:0/2704447050 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17400cb00 con 0x7fc1840adc70 2023-10-30T12:09:42.263 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc172ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:42.263 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc172ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:42.263 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc172ffd640 10 monclient: dump: 2023-10-30T12:09:42.263 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:42.263 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:42.263 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:42.264 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:42.264 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:42.264 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:42.264 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:42.265 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:42.265 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:42.265 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:42.265 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc188a4b640 5 monclient: authenticate success, global_id 4163 2023-10-30T12:09:42.265 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc180d99640 4 set_mon_vals no callback set 2023-10-30T12:09:42.266 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc188a4b640 10 monclient: _renew_subs 2023-10-30T12:09:42.266 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc188a4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:09:42.266 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc188a4b640 1 -- 172.21.15.3:0/2704447050 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1840abe40 con 0x7fc1840adc70 2023-10-30T12:09:42.266 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc188a4b640 10 client.4163.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:42.266 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc188a4b640 10 monclient: _renew_subs 2023-10-30T12:09:42.267 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc188a4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:09:42.267 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc188a4b640 1 -- 172.21.15.3:0/2704447050 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1841d7090 con 0x7fc1840adc70 2023-10-30T12:09:42.267 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.233+0000 7fc188a4b640 1 librados: init done 2023-10-30T12:09:42.267 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.234+0000 7fc188a4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:42.267 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.234+0000 7fc188a4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:42.268 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.234+0000 7fc188a4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:09:42.268 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.234+0000 7fc188a4b640 1 -- 172.21.15.3:0/2704447050 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc18404e9b0 con 0x7fc1840adc70 2023-10-30T12:09:42.268 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.256+0000 7fc172ffd640 1 -- 172.21.15.3:0/2704447050 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7fc17400cc70 con 0x7fc1840adc70 2023-10-30T12:09:42.268 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.256+0000 7fc172ffd640 1 --2- 172.21.15.3:0/2704447050 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc16406c730 0x7fc16406ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:42.268 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.257+0000 7fc181d9b640 1 --2- 172.21.15.3:0/2704447050 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc16406c730 0x7fc16406ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:42.269 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.257+0000 7fc181d9b640 10 monclient: get_auth_request con 0x7fc16406c730 auth_method 0 2023-10-30T12:09:42.269 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.257+0000 7fc172ffd640 10 client.4163.objecter ms_handle_connect 0x7fc16406c730 2023-10-30T12:09:42.269 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.257+0000 7fc181d9b640 1 --2- 172.21.15.3:0/2704447050 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc16406c730 0x7fc16406ebf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc17802b810 tx=0x7fc178008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:42.354 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.352+0000 7fc172ffd640 1 -- 172.21.15.3:0/2704447050 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc1740584b0 con 0x7fc1840adc70 2023-10-30T12:09:42.354 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.352+0000 7fc172ffd640 10 client.4163.objecter ms_dispatch 0x7fc184000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T12:09:42.354 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.352+0000 7fc172ffd640 3 client.4163.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T12:09:42.354 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.352+0000 7fc172ffd640 3 client.4163.objecter handle_osd_map decoding full epoch 10 2023-10-30T12:09:42.355 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.352+0000 7fc172ffd640 20 client.4163.objecter dump_active .. 0 homeless 2023-10-30T12:09:42.610 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.609+0000 7fc172ffd640 1 -- 172.21.15.3:0/2704447050 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc17405c550 con 0x7fc1840adc70 2023-10-30T12:09:42.611 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.609+0000 7fc172ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:42.611 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.609+0000 7fc172ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:42.727 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:17.688 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:42.878 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.877+0000 7fc188a4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:42.879 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.877+0000 7fc188a4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:42.879 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.877+0000 7fc188a4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:09:42.879 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.877+0000 7fc188a4b640 1 -- 172.21.15.3:0/2704447050 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1840abe40 con 0x7fc1840adc70 2023-10-30T12:09:42.896 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:17.857 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:42.939 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.937+0000 7fc172ffd640 1 -- 172.21.15.3:0/2704447050 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fc174060700 con 0x7fc1840adc70 2023-10-30T12:09:42.939 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.937+0000 7fc172ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:42.939 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.937+0000 7fc172ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:42.940 INFO:teuthology.misc.health.smithi003.stdout: 2023-10-30T12:09:42.940 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"9bae44ae-7379-4ab2-a454-4d550607110c","created":"2023-10-30T12:09:17.680340+0000","modified":"2023-10-30T12:09:24.800567+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:09:24.800567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69005df2-28b6-4bb2-b3e6-79df0d403c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6466ea5c-d74a-4709-a83a-f5d4fe17157e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"69ec5657-9972-4f4b-8012-83886eb47429","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3dad24-4aeb-47a3-ad1d-d13c77cafad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d38d9d7f-b0de-48ab-a850-499c6a0a4364","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b39d2b0a-84ef-4909-bac7-47f146305a79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0a9d81ac-5a14-4fef-877a-982594787dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:09:42.941 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.939+0000 7fc188a4b640 10 librados: watch_flush enter 2023-10-30T12:09:42.941 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.939+0000 7fc188a4b640 10 librados: watch_flush exit 2023-10-30T12:09:42.941 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.939+0000 7fc188a4b640 20 client.4163.objecter shutdown clearing up homeless session... 2023-10-30T12:09:42.942 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.939+0000 7fc188a4b640 10 client.4163.objecter successfully canceled tick 2023-10-30T12:09:42.942 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.939+0000 7fc188a4b640 1 -- 172.21.15.3:0/2704447050 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc16406c730 msgr2=0x7fc16406ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:42.942 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.939+0000 7fc188a4b640 1 --2- 172.21.15.3:0/2704447050 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc16406c730 0x7fc16406ebf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc17802b810 tx=0x7fc178008030 comp rx=0 tx=0).stop 2023-10-30T12:09:42.942 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.939+0000 7fc188a4b640 10 monclient: shutdown 2023-10-30T12:09:42.942 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.939+0000 7fc188a4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:42.943 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.939+0000 7fc188a4b640 1 -- 172.21.15.3:0/2704447050 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1840adc70 msgr2=0x7fc1840accf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:42.943 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.939+0000 7fc188a4b640 1 --2- 172.21.15.3:0/2704447050 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1840adc70 0x7fc1840accf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc17400e870 tx=0x7fc17400ed30 comp rx=0 tx=0).stop 2023-10-30T12:09:42.944 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.940+0000 7fc188a4b640 1 -- 172.21.15.3:0/2704447050 shutdown_connections 2023-10-30T12:09:42.944 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.940+0000 7fc188a4b640 1 --2- 172.21.15.3:0/2704447050 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc16406c730 0x7fc16406ebf0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:42.944 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.940+0000 7fc188a4b640 1 --2- 172.21.15.3:0/2704447050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1840b4350 0x7fc18411f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:42.944 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.940+0000 7fc188a4b640 1 --2- 172.21.15.3:0/2704447050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1840b0d50 0x7fc1840aa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:42.945 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.940+0000 7fc188a4b640 1 --2- 172.21.15.3:0/2704447050 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1840adc70 0x7fc1840accf0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:42.945 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.940+0000 7fc188a4b640 1 -- 172.21.15.3:0/2704447050 >> 172.21.15.3:0/2704447050 conn(0x7fc184060bf0 msgr2=0x7fc18413cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:42.945 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.941+0000 7fc188a4b640 1 -- 172.21.15.3:0/2704447050 shutdown_connections 2023-10-30T12:09:42.945 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.941+0000 7fc188a4b640 1 -- 172.21.15.3:0/2704447050 wait complete. 2023-10-30T12:09:42.946 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:42.941+0000 7fc188a4b640 1 librados: shutdown 2023-10-30T12:09:42.952 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T12:09:42.963 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:17.914 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:42.971 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:17.928 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:43.458 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:18.411 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:43.673 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:18.607 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:43.732 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:18.689 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:43.780 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:18.712 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:48.399 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:23.359 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:48.402 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:23.363 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:48.581 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:23.538 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:48.585 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:23.542 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:48.779 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:23.731 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:48.783 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:23.734 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:48.851 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:23.812 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:48.855 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:23.816 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:49.008 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:23.961 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:49.012 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:23.965 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:49.211 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:24.168 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:49.216 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:24.172 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:49.312 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:24.245 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:49.317 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:24.249 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:49.321 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:24.255 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:49.326 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:24.259 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:49.757 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:09:49.853 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.851+0000 7fee10e4b640 10 monclient: get_monmap_and_config 2023-10-30T12:09:49.853 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.851+0000 7fee10e4b640 10 monclient: build_initial_monmap 2023-10-30T12:09:49.854 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.851+0000 7fee10e4b640 10 monclient: monmap: 2023-10-30T12:09:49.854 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2023-10-30T12:09:49.854 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:49.854 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:09:49.852900+0000 2023-10-30T12:09:49.855 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:09:49.852900+0000 2023-10-30T12:09:49.855 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:49.855 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:49.855 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:09:49.855 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:09:49.856 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:09:49.856 INFO:teuthology.misc.health.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:09:49.856 INFO:teuthology.misc.health.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:09:49.856 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:49.858 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.852+0000 7fee10e4b640 1 Processor -- start 2023-10-30T12:09:49.859 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.852+0000 7fee10e4b640 1 -- start start 2023-10-30T12:09:49.859 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.852+0000 7fee10e4b640 10 monclient: init 2023-10-30T12:09:49.859 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.853+0000 7fee10e4b640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:49.859 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.853+0000 7fee10e4b640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-10-30T12:09:49.859 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.853+0000 7fee10e4b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fee0c0a8ca0 0x7fee0c141f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:49.860 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.853+0000 7fee10e4b640 10 monclient(hunting): picked mon.noname-c con 0x7fee0c0a8ca0 addr v2:172.21.15.92:3300/0 2023-10-30T12:09:49.860 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.853+0000 7fee10e4b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee0c142450 addr v1:172.21.15.3:6789/0 2023-10-30T12:09:49.860 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.853+0000 7fee10e4b640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fee0c0a9080 0x7fee0c14c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:49.860 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.853+0000 7fee10e4b640 10 monclient(hunting): picked mon.noname-a con 0x7fee0c0a9080 addr v2:172.21.15.3:3300/0 2023-10-30T12:09:49.861 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.853+0000 7fee10e4b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c0a9540 con 0x7fee0c142450 2023-10-30T12:09:49.861 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.853+0000 7fee10e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:49.861 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.853+0000 7fee10e4b640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee0c067a30 con 0x7fee0c0a9080 2023-10-30T12:09:49.861 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.853+0000 7fee10e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:49.861 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.853+0000 7fee10e4b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee0c0aa770 con 0x7fee0c0a8ca0 2023-10-30T12:09:49.862 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.853+0000 7fee10e4b640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:49.862 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.853+0000 7fee10e4b640 10 monclient(hunting): authenticate will time out at 1381.067705s 2023-10-30T12:09:49.862 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.854+0000 7fee0a59c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fee0c0a8ca0 0x7fee0c141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:49.862 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.854+0000 7fee0ad9d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fee0c0a9080 0x7fee0c14c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:49.862 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.854+0000 7fee0ad9d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fee0c0a9080 0x7fee0c14c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49918/0 (socket says 172.21.15.3:49918) 2023-10-30T12:09:49.863 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.854+0000 7fee0ad9d640 1 -- 172.21.15.3:0/1049115247 learned_addr learned my addr 172.21.15.3:0/1049115247 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:09:49.863 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.854+0000 7fee0ad9d640 10 monclient(hunting): get_auth_request con 0x7fee0c0a9080 auth_method 0 2023-10-30T12:09:49.863 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.854+0000 7fee0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:49.863 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.854+0000 7fee0ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:49.863 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.854+0000 7fee0ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:49.863 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.855+0000 7fee0a59c640 10 monclient(hunting): get_auth_request con 0x7fee0c0a8ca0 auth_method 0 2023-10-30T12:09:49.864 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.855+0000 7fee0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:49.864 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.855+0000 7fee0a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:49.864 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.855+0000 7fee0a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:49.864 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.855+0000 7fee0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:49.864 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.855+0000 7fee0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:49.865 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.855+0000 7fee0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:49.865 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.855+0000 7fee0ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:49.865 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.855+0000 7fee0ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:49.865 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.855+0000 7fee0ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:49.870 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.858+0000 7fee0a59c640 10 monclient(hunting): handle_auth_done global_id 4207 payload 274 2023-10-30T12:09:49.870 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.858+0000 7fee0a59c640 1 -- 172.21.15.3:0/1049115247 >> v2:172.21.15.3:3300/0 conn(0x7fee0c0a9080 msgr2=0x7fee0c14c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:49.870 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.858+0000 7fee0a59c640 1 --2- 172.21.15.3:0/1049115247 >> v2:172.21.15.3:3300/0 conn(0x7fee0c0a9080 0x7fee0c14c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:49.870 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.858+0000 7fee0a59c640 1 -- 172.21.15.3:0/1049115247 >> v1:172.21.15.3:6789/0 conn(0x7fee0c142450 legacy=0x7fee0c13ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:49.871 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.858+0000 7fee0a59c640 10 monclient: _finish_hunting 0 2023-10-30T12:09:49.871 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.858+0000 7fee0a59c640 1 monclient: found mon.noname-c 2023-10-30T12:09:49.871 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.858+0000 7fee0a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:49.871 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.858+0000 7fee0a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:09:49.871 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.858+0000 7fee0a59c640 1 -- 172.21.15.3:0/1049115247 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c0b39e0 con 0x7fee0c0a8ca0 2023-10-30T12:09:49.872 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.860+0000 7fee0a59c640 1 --2- 172.21.15.3:0/1049115247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c0a8ca0 0x7fee0c141f10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0b7100 tx=0x7fedfc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e69d2a95164f887 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:49.872 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.865+0000 7fee0959a640 1 -- 172.21.15.3:0/1049115247 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc0315a0 con 0x7fee0c0a8ca0 2023-10-30T12:09:49.872 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.865+0000 7fee0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:49.872 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.865+0000 7fee0959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:49.872 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.865+0000 7fee0959a640 10 monclient: dump: 2023-10-30T12:09:49.873 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:49.873 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:49.873 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:49.873 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:49.873 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:49.874 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:49.874 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:49.874 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:49.874 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:49.874 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:49.875 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.865+0000 7fee0959a640 10 monclient: _finish_auth 0 2023-10-30T12:09:49.875 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.865+0000 7fee0959a640 10 monclient: _check_auth_tickets 2023-10-30T12:09:49.875 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.865+0000 7fee0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:49.875 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.865+0000 7fee10e4b640 5 monclient: authenticate success, global_id 4207 2023-10-30T12:09:49.875 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.865+0000 7fee10e4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:49.875 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.865+0000 7fee0959a640 1 -- 172.21.15.3:0/1049115247 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc031710 con 0x7fee0c0a8ca0 2023-10-30T12:09:49.876 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.865+0000 7fee0959a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:49.876 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.865+0000 7fee10e4b640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:49.876 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.865+0000 7fee10e4b640 4 set_mon_vals no callback set 2023-10-30T12:09:49.876 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.866+0000 7fee0959a640 1 -- 172.21.15.3:0/1049115247 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc0315a0 con 0x7fee0c0a8ca0 2023-10-30T12:09:49.876 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.866+0000 7fee0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:49.877 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.866+0000 7fee0959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:49.877 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.866+0000 7fee0959a640 10 monclient: dump: 2023-10-30T12:09:49.877 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:49.877 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:49.877 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:49.877 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:49.878 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:49.878 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:49.878 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:49.878 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:49.878 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:49.879 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:49.879 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.866+0000 7fee10e4b640 10 monclient: shutdown 2023-10-30T12:09:49.879 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.866+0000 7fee10e4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:49.879 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.866+0000 7fee10e4b640 1 -- 172.21.15.3:0/1049115247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c0a8ca0 msgr2=0x7fee0c141f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:49.879 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.866+0000 7fee10e4b640 1 --2- 172.21.15.3:0/1049115247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c0a8ca0 0x7fee0c141f10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0b7100 tx=0x7fedfc02b840 comp rx=0 tx=0).stop 2023-10-30T12:09:49.880 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.866+0000 7fee10e4b640 1 -- 172.21.15.3:0/1049115247 shutdown_connections 2023-10-30T12:09:49.880 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.866+0000 7fee10e4b640 1 --2- 172.21.15.3:0/1049115247 >> v2:172.21.15.3:3300/0 conn(0x7fee0c0a9080 0x7fee0c14c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:49.880 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.866+0000 7fee10e4b640 1 --2- 172.21.15.3:0/1049115247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c0a8ca0 0x7fee0c141f10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:49.880 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.867+0000 7fee10e4b640 1 -- 172.21.15.3:0/1049115247 >> 172.21.15.3:0/1049115247 conn(0x7fee0c0b71a0 msgr2=0x7fee0c0b75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:49.880 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.867+0000 7fee10e4b640 1 -- 172.21.15.3:0/1049115247 shutdown_connections 2023-10-30T12:09:49.880 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.867+0000 7fee10e4b640 1 -- 172.21.15.3:0/1049115247 wait complete. 2023-10-30T12:09:49.881 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.868+0000 7fee10e4b640 10 monclient: build_initial_monmap 2023-10-30T12:09:49.881 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.868+0000 7fee10e4b640 10 monclient: monmap: 2023-10-30T12:09:49.881 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2023-10-30T12:09:49.881 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:49.881 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2023-10-30T12:09:49.881 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2023-10-30T12:09:49.882 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:49.882 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2023-10-30T12:09:49.882 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:09:49.882 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:09:49.882 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:09:49.883 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:49.883 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.868+0000 7fee10e4b640 1 librados: starting msgr at 2023-10-30T12:09:49.883 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.868+0000 7fee10e4b640 1 librados: starting objecter 2023-10-30T12:09:49.883 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.868+0000 7fee10e4b640 1 Processor -- start 2023-10-30T12:09:49.883 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.868+0000 7fee10e4b640 1 -- start start 2023-10-30T12:09:49.884 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.868+0000 7fee10e4b640 1 librados: setting wanted keys 2023-10-30T12:09:49.884 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.868+0000 7fee10e4b640 1 librados: calling monclient init 2023-10-30T12:09:49.884 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.868+0000 7fee10e4b640 10 monclient: init 2023-10-30T12:09:49.889 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.869+0000 7fee10e4b640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:49.889 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.869+0000 7fee10e4b640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:09:49.890 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.869+0000 7fee10e4b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee0c0a8ca0 0x7fee0c0b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:49.890 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.869+0000 7fee10e4b640 10 monclient(hunting): picked mon.noname-a con 0x7fee0c0a8ca0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:49.890 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.869+0000 7fee10e4b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c0a9080 0x7fee0c0abb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:49.890 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.870+0000 7fee10e4b640 10 monclient(hunting): picked mon.noname-b con 0x7fee0c0a9080 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:49.890 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.870+0000 7fee10e4b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee0c0ad830 0x7fee0c0b4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:49.891 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.870+0000 7fee10e4b640 10 monclient(hunting): picked mon.noname-c con 0x7fee0c0ad830 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:49.891 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.870+0000 7fee10e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:49.891 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.870+0000 7fee10e4b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c0a84f0 con 0x7fee0c0a8ca0 2023-10-30T12:09:49.891 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.870+0000 7fee10e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:49.891 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.870+0000 7fee10e4b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c0ac330 con 0x7fee0c0a9080 2023-10-30T12:09:49.891 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.870+0000 7fee10e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:49.892 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.870+0000 7fee10e4b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c0acef0 con 0x7fee0c0ad830 2023-10-30T12:09:49.892 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.870+0000 7fee10e4b640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:49.892 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.870+0000 7fee10e4b640 10 monclient(hunting): authenticate will time out at 1381.084798s 2023-10-30T12:09:49.892 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.870+0000 7fee0a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee0c0a8ca0 0x7fee0c0b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:49.892 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.871+0000 7fee09d9b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c0a9080 0x7fee0c0abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:49.893 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.871+0000 7fee0a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee0c0a8ca0 0x7fee0c0b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49922/0 (socket says 172.21.15.3:49922) 2023-10-30T12:09:49.893 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.871+0000 7fee0a59c640 1 -- 172.21.15.3:0/3314647837 learned_addr learned my addr 172.21.15.3:0/3314647837 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:09:49.893 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.871+0000 7fee0a59c640 10 monclient(hunting): get_auth_request con 0x7fee0c0a8ca0 auth_method 0 2023-10-30T12:09:49.893 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.871+0000 7fee0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:49.893 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.871+0000 7fee0a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:49.894 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.871+0000 7fee0a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:49.894 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.871+0000 7fee0ad9d640 1 --2- 172.21.15.3:0/3314647837 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee0c0ad830 0x7fee0c0b4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:49.894 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.871+0000 7fee09d9b640 10 monclient(hunting): get_auth_request con 0x7fee0c0a9080 auth_method 0 2023-10-30T12:09:49.894 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.871+0000 7fee09d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:49.894 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.871+0000 7fee09d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:49.895 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.871+0000 7fee09d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:49.895 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.871+0000 7fee0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:49.895 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.871+0000 7fee0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:49.895 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.871+0000 7fee0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:49.896 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.872+0000 7fee0ad9d640 10 monclient(hunting): get_auth_request con 0x7fee0c0ad830 auth_method 0 2023-10-30T12:09:49.896 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.872+0000 7fee0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:49.896 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.872+0000 7fee0ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:49.896 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.872+0000 7fee0ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:49.896 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.872+0000 7fee09d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:49.896 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.872+0000 7fee09d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:49.897 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.872+0000 7fee09d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:49.897 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.872+0000 7fee0ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:49.897 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.872+0000 7fee0ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:49.897 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.872+0000 7fee0ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:49.897 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.874+0000 7fee0a59c640 10 monclient(hunting): handle_auth_done global_id 4230 payload 824 2023-10-30T12:09:49.898 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.874+0000 7fee0a59c640 1 -- 172.21.15.3:0/3314647837 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee0c0ad830 msgr2=0x7fee0c0b4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:49.898 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.874+0000 7fee0a59c640 1 --2- 172.21.15.3:0/3314647837 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee0c0ad830 0x7fee0c0b4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:49.898 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.875+0000 7fee0a59c640 1 -- 172.21.15.3:0/3314647837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c0a9080 msgr2=0x7fee0c0abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:49.898 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.875+0000 7fee0a59c640 1 --2- 172.21.15.3:0/3314647837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c0a9080 0x7fee0c0abb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:49.898 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.875+0000 7fee0a59c640 10 monclient: _finish_hunting 0 2023-10-30T12:09:49.899 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.875+0000 7fee0a59c640 1 monclient: found mon.noname-a 2023-10-30T12:09:49.899 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.875+0000 7fee0a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:49.899 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.875+0000 7fee0a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:09:49.899 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.875+0000 7fee0a59c640 1 -- 172.21.15.3:0/3314647837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c0ac0f0 con 0x7fee0c0a8ca0 2023-10-30T12:09:49.899 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.878+0000 7fee0a59c640 1 --2- 172.21.15.3:0/3314647837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee0c0a8ca0 0x7fee0c0b12a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fedfc034040 tx=0x7fedfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:49.900 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.878+0000 7fedfaffd640 10 client.?.objecter ms_handle_connect 0x7fee0c0a8ca0 2023-10-30T12:09:49.900 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.878+0000 7fedfaffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:49.900 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fedfaffd640 1 -- 172.21.15.3:0/3314647837 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc040070 con 0x7fee0c0a8ca0 2023-10-30T12:09:49.900 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fedfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:49.900 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fedfaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:49.901 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fedfaffd640 10 monclient: dump: 2023-10-30T12:09:49.901 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:49.901 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:49.901 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:49.901 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:49.901 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:49.902 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:49.902 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:49.902 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:49.902 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:49.902 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:49.903 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fedfaffd640 10 monclient: _finish_auth 0 2023-10-30T12:09:49.903 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fedfaffd640 10 monclient: _check_auth_tickets 2023-10-30T12:09:49.903 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fedfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:49.903 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fedfaffd640 1 -- 172.21.15.3:0/3314647837 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc003a20 con 0x7fee0c0a8ca0 2023-10-30T12:09:49.903 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fedfaffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:49.904 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fedfaffd640 1 -- 172.21.15.3:0/3314647837 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc003de0 con 0x7fee0c0a8ca0 2023-10-30T12:09:49.904 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fedfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:49.904 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fedfaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:49.904 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fedfaffd640 10 monclient: dump: 2023-10-30T12:09:49.904 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:49.905 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:49.905 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:49.905 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:49.905 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:49.905 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:49.906 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:49.906 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:49.906 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:49.906 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:49.906 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fee10e4b640 5 monclient: authenticate success, global_id 4230 2023-10-30T12:09:49.907 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fee10e4b640 10 monclient: _renew_subs 2023-10-30T12:09:49.907 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fee10e4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:09:49.907 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fee10e4b640 1 -- 172.21.15.3:0/3314647837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0c0ad190 con 0x7fee0c0a8ca0 2023-10-30T12:09:49.907 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fee08d99640 4 set_mon_vals no callback set 2023-10-30T12:09:49.907 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fee10e4b640 10 client.4230.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:49.908 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fee10e4b640 10 monclient: _renew_subs 2023-10-30T12:09:49.908 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fee10e4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:09:49.908 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fee10e4b640 1 -- 172.21.15.3:0/3314647837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0c0b54e0 con 0x7fee0c0a8ca0 2023-10-30T12:09:49.908 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.881+0000 7fee10e4b640 1 librados: init done 2023-10-30T12:09:49.908 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.883+0000 7fee10e4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:49.909 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.883+0000 7fee10e4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:49.909 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.883+0000 7fee10e4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:09:49.909 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.883+0000 7fee10e4b640 1 -- 172.21.15.3:0/3314647837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0c0b5710 con 0x7fee0c0a8ca0 2023-10-30T12:09:49.910 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.903+0000 7fedfaffd640 1 -- 172.21.15.3:0/3314647837 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7fedfc047480 con 0x7fee0c0a8ca0 2023-10-30T12:09:49.911 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.903+0000 7fedfaffd640 1 --2- 172.21.15.3:0/3314647837 >> v2:172.21.15.92:6800/1750155735 conn(0x7fedec06c780 0x7fedec06ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:49.914 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.904+0000 7fee09d9b640 1 --2- 172.21.15.3:0/3314647837 >> v2:172.21.15.92:6800/1750155735 conn(0x7fedec06c780 0x7fedec06ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:49.914 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.904+0000 7fee09d9b640 10 monclient: get_auth_request con 0x7fedec06c780 auth_method 0 2023-10-30T12:09:49.914 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.904+0000 7fee09d9b640 1 --2- 172.21.15.3:0/3314647837 >> v2:172.21.15.92:6800/1750155735 conn(0x7fedec06c780 0x7fedec06ec40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0b6cb0 tx=0x7fee00009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:49.915 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.904+0000 7fedfaffd640 10 client.4230.objecter ms_handle_connect 0x7fedec06c780 2023-10-30T12:09:49.915 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.905+0000 7fedfaffd640 1 -- 172.21.15.3:0/3314647837 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fedfc07a820 con 0x7fee0c0a8ca0 2023-10-30T12:09:49.915 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.905+0000 7fedfaffd640 10 client.4230.objecter ms_dispatch 0x7fee0c000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T12:09:49.915 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.905+0000 7fedfaffd640 3 client.4230.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T12:09:49.915 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.905+0000 7fedfaffd640 3 client.4230.objecter handle_osd_map decoding full epoch 10 2023-10-30T12:09:49.915 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:49.905+0000 7fedfaffd640 20 client.4230.objecter dump_active .. 0 homeless 2023-10-30T12:09:50.321 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.320+0000 7fedfaffd640 1 -- 172.21.15.3:0/3314647837 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fedfc037070 con 0x7fee0c0a8ca0 2023-10-30T12:09:50.322 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.320+0000 7fedfaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:50.322 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.320+0000 7fedfaffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:50.603 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.601+0000 7fee10e4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:50.603 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.601+0000 7fee10e4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:50.603 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.601+0000 7fee10e4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:09:50.603 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.601+0000 7fee10e4b640 1 -- 172.21.15.3:0/3314647837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee0c0ac0f0 con 0x7fee0c0a8ca0 2023-10-30T12:09:50.745 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.743+0000 7fedfaffd640 1 -- 172.21.15.3:0/3314647837 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fedfc07e8c0 con 0x7fee0c0a8ca0 2023-10-30T12:09:50.745 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.743+0000 7fedfaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:50.746 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.743+0000 7fedfaffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:50.746 INFO:teuthology.misc.health.smithi003.stdout: 2023-10-30T12:09:50.746 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"9bae44ae-7379-4ab2-a454-4d550607110c","created":"2023-10-30T12:09:17.680340+0000","modified":"2023-10-30T12:09:24.800567+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:09:24.800567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69005df2-28b6-4bb2-b3e6-79df0d403c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6466ea5c-d74a-4709-a83a-f5d4fe17157e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"69ec5657-9972-4f4b-8012-83886eb47429","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3dad24-4aeb-47a3-ad1d-d13c77cafad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d38d9d7f-b0de-48ab-a850-499c6a0a4364","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b39d2b0a-84ef-4909-bac7-47f146305a79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0a9d81ac-5a14-4fef-877a-982594787dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:09:50.748 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 10 librados: watch_flush enter 2023-10-30T12:09:50.748 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 10 librados: watch_flush exit 2023-10-30T12:09:50.748 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 20 client.4230.objecter shutdown clearing up homeless session... 2023-10-30T12:09:50.748 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 10 client.4230.objecter successfully canceled tick 2023-10-30T12:09:50.749 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 1 -- 172.21.15.3:0/3314647837 >> v2:172.21.15.92:6800/1750155735 conn(0x7fedec06c780 msgr2=0x7fedec06ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:50.749 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 1 --2- 172.21.15.3:0/3314647837 >> v2:172.21.15.92:6800/1750155735 conn(0x7fedec06c780 0x7fedec06ec40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0b6cb0 tx=0x7fee00009040 comp rx=0 tx=0).stop 2023-10-30T12:09:50.749 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 10 monclient: shutdown 2023-10-30T12:09:50.749 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:50.749 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 1 -- 172.21.15.3:0/3314647837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee0c0a8ca0 msgr2=0x7fee0c0b12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:50.750 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 1 --2- 172.21.15.3:0/3314647837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee0c0a8ca0 0x7fee0c0b12a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fedfc034040 tx=0x7fedfc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:09:50.750 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 1 -- 172.21.15.3:0/3314647837 shutdown_connections 2023-10-30T12:09:50.750 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 1 --2- 172.21.15.3:0/3314647837 >> v2:172.21.15.92:6800/1750155735 conn(0x7fedec06c780 0x7fedec06ec40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:50.750 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 1 --2- 172.21.15.3:0/3314647837 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee0c0ad830 0x7fee0c0b4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:50.750 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 1 --2- 172.21.15.3:0/3314647837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c0a9080 0x7fee0c0abb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:50.751 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 1 --2- 172.21.15.3:0/3314647837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee0c0a8ca0 0x7fee0c0b12a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:50.751 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 1 -- 172.21.15.3:0/3314647837 >> 172.21.15.3:0/3314647837 conn(0x7fee0c0b71a0 msgr2=0x7fee0c13a340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:50.751 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.746+0000 7fee10e4b640 1 -- 172.21.15.3:0/3314647837 shutdown_connections 2023-10-30T12:09:50.751 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.747+0000 7fee10e4b640 1 -- 172.21.15.3:0/3314647837 wait complete. 2023-10-30T12:09:50.751 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:50.747+0000 7fee10e4b640 1 librados: shutdown 2023-10-30T12:09:50.762 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T12:09:52.354 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:27.315 86369== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:52.637 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:27.594 86371== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:52.760 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:27.721 86370== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:52.894 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:27.845 86372== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:53.418 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:28.371 86747== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:53.702 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:28.659 86746== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:53.731 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:28.664 86748== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:53.754 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:28.686 86749== Warning: unimplemented fcntl command: 1038 2023-10-30T12:09:57.569 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:09:57.715 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.713+0000 7feb5d80e640 10 monclient: get_monmap_and_config 2023-10-30T12:09:57.715 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.713+0000 7feb5d80e640 10 monclient: build_initial_monmap 2023-10-30T12:09:57.715 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.713+0000 7feb5d80e640 10 monclient: monmap: 2023-10-30T12:09:57.716 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2023-10-30T12:09:57.716 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:57.716 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:09:57.714522+0000 2023-10-30T12:09:57.716 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:09:57.714522+0000 2023-10-30T12:09:57.716 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:57.717 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:57.717 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:09:57.717 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:09:57.717 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:09:57.717 INFO:teuthology.misc.health.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:09:57.717 INFO:teuthology.misc.health.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:09:57.718 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:57.719 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.714+0000 7feb5d80e640 1 Processor -- start 2023-10-30T12:09:57.720 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.714+0000 7feb5d80e640 1 -- start start 2023-10-30T12:09:57.720 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.714+0000 7feb5d80e640 10 monclient: init 2023-10-30T12:09:57.720 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.715+0000 7feb5d80e640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:57.720 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.715+0000 7feb5d80e640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2023-10-30T12:09:57.720 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.715+0000 7feb5d80e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb580b4350 addr v1:172.21.15.92:6789/0 2023-10-30T12:09:57.721 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.715+0000 7feb5d80e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb580adc70 addr v1:172.21.15.3:6789/0 2023-10-30T12:09:57.721 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.715+0000 7feb5d80e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb5813ebb0 0x7feb5813f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:57.721 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.715+0000 7feb5d80e640 10 monclient(hunting): picked mon.noname-b con 0x7feb5813ebb0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:57.721 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.715+0000 7feb5d80e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb580ae950 con 0x7feb580adc70 2023-10-30T12:09:57.721 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.715+0000 7feb5d80e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb580b0f20 con 0x7feb580b4350 2023-10-30T12:09:57.722 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.715+0000 7feb5d80e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:57.722 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.715+0000 7feb5d80e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb58067a10 con 0x7feb5813ebb0 2023-10-30T12:09:57.722 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.715+0000 7feb5d80e640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:57.722 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.715+0000 7feb5d80e640 10 monclient(hunting): authenticate will time out at 1388.929672s 2023-10-30T12:09:57.722 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.716+0000 7feb567fc640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7feb580adc70 0x7feb580b0810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56172/0 (socket says 172.21.15.3:56172) 2023-10-30T12:09:57.723 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.716+0000 7feb567fc640 1 -- 172.21.15.3:0/3445580525 learned_addr learned my addr 172.21.15.3:0/3445580525 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:09:57.723 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.717+0000 7feb577fe640 1 --2- 172.21.15.3:0/3445580525 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb5813ebb0 0x7feb5813f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:57.723 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.717+0000 7feb577fe640 10 monclient(hunting): get_auth_request con 0x7feb5813ebb0 auth_method 0 2023-10-30T12:09:57.723 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.717+0000 7feb577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:57.723 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.717+0000 7feb577fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:57.724 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.717+0000 7feb577fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:57.724 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.718+0000 7feb577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:57.724 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.718+0000 7feb577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:57.724 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.718+0000 7feb577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:57.725 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.720+0000 7feb55ffb640 1 -- 172.21.15.3:0/3445580525 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405175498 0 0) 0x7feb48001110 con 0x7feb580b4350 2023-10-30T12:09:57.726 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.720+0000 7feb55ffb640 10 monclient(hunting): _negotiate 2023-10-30T12:09:57.726 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.720+0000 7feb55ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:57.726 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.720+0000 7feb55ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:57.726 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.720+0000 7feb55ffb640 10 monclient(hunting): my global_id is 4213 2023-10-30T12:09:57.726 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.720+0000 7feb55ffb640 1 -- 172.21.15.3:0/3445580525 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb40003520 con 0x7feb580b4350 2023-10-30T12:09:57.727 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.720+0000 7feb577fe640 10 monclient(hunting): handle_auth_done global_id 4169 payload 274 2023-10-30T12:09:57.727 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.720+0000 7feb577fe640 1 -- 172.21.15.3:0/3445580525 >> v1:172.21.15.92:6789/0 conn(0x7feb580b4350 legacy=0x7feb580aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:57.727 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.720+0000 7feb577fe640 1 -- 172.21.15.3:0/3445580525 >> v1:172.21.15.3:6789/0 conn(0x7feb580adc70 legacy=0x7feb580b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:57.727 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.720+0000 7feb577fe640 10 monclient: _finish_hunting 0 2023-10-30T12:09:57.727 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.720+0000 7feb577fe640 1 monclient: found mon.noname-b 2023-10-30T12:09:57.728 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.720+0000 7feb577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:57.728 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.720+0000 7feb577fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:09:57.728 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.720+0000 7feb577fe640 1 -- 172.21.15.3:0/3445580525 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb580b3990 con 0x7feb5813ebb0 2023-10-30T12:09:57.728 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.724+0000 7feb577fe640 1 --2- 172.21.15.3:0/3445580525 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb5813ebb0 0x7feb5813f010 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feb50009dc0 tx=0x7feb5002be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c33b24c9b2ef177e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:57.733 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb55ffb640 1 -- 172.21.15.3:0/3445580525 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5002cc30 con 0x7feb5813ebb0 2023-10-30T12:09:57.733 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:57.733 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb55ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:57.733 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb55ffb640 10 monclient: dump: 2023-10-30T12:09:57.733 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:57.734 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:57.734 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:57.734 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:57.734 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:57.734 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:57.735 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:57.735 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:57.735 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:57.735 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:57.735 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb55ffb640 10 monclient: _finish_auth 0 2023-10-30T12:09:57.735 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb55ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:09:57.736 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:57.736 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb55ffb640 1 -- 172.21.15.3:0/3445580525 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb5002cda0 con 0x7feb5813ebb0 2023-10-30T12:09:57.736 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb55ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:57.736 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb55ffb640 1 -- 172.21.15.3:0/3445580525 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb50034660 con 0x7feb5813ebb0 2023-10-30T12:09:57.736 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb5d80e640 5 monclient: authenticate success, global_id 4169 2023-10-30T12:09:57.737 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb5d80e640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:57.737 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb5d80e640 4 set_mon_vals no callback set 2023-10-30T12:09:57.737 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:57.737 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb55ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:57.737 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.731+0000 7feb55ffb640 10 monclient: dump: 2023-10-30T12:09:57.738 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:57.738 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:57.738 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:57.738 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:57.738 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:57.739 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:57.739 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:57.739 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:57.739 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:57.739 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:57.745 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.732+0000 7feb5d80e640 10 monclient: shutdown 2023-10-30T12:09:57.745 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.732+0000 7feb5d80e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:57.745 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.732+0000 7feb5d80e640 1 -- 172.21.15.3:0/3445580525 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb5813ebb0 msgr2=0x7feb5813f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:57.746 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.732+0000 7feb5d80e640 1 --2- 172.21.15.3:0/3445580525 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb5813ebb0 0x7feb5813f010 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feb50009dc0 tx=0x7feb5002be80 comp rx=0 tx=0).stop 2023-10-30T12:09:57.746 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.732+0000 7feb5d80e640 1 -- 172.21.15.3:0/3445580525 shutdown_connections 2023-10-30T12:09:57.746 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.732+0000 7feb5d80e640 1 --2- 172.21.15.3:0/3445580525 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb5813ebb0 0x7feb5813f010 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:57.746 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.732+0000 7feb5d80e640 1 -- 172.21.15.3:0/3445580525 >> 172.21.15.3:0/3445580525 conn(0x7feb58060bf0 msgr2=0x7feb58061000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:57.746 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.732+0000 7feb5d80e640 1 -- 172.21.15.3:0/3445580525 shutdown_connections 2023-10-30T12:09:57.747 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.732+0000 7feb5d80e640 1 -- 172.21.15.3:0/3445580525 wait complete. 2023-10-30T12:09:57.747 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.733+0000 7feb5d80e640 10 monclient: build_initial_monmap 2023-10-30T12:09:57.747 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.733+0000 7feb5d80e640 10 monclient: monmap: 2023-10-30T12:09:57.747 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2023-10-30T12:09:57.747 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:57.747 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2023-10-30T12:09:57.748 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2023-10-30T12:09:57.748 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:57.748 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2023-10-30T12:09:57.748 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:09:57.748 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:09:57.749 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:09:57.749 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:57.749 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.734+0000 7feb5d80e640 1 librados: starting msgr at 2023-10-30T12:09:57.749 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.734+0000 7feb5d80e640 1 librados: starting objecter 2023-10-30T12:09:57.749 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.734+0000 7feb5d80e640 1 Processor -- start 2023-10-30T12:09:57.750 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.734+0000 7feb5d80e640 1 -- start start 2023-10-30T12:09:57.750 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.734+0000 7feb5d80e640 1 librados: setting wanted keys 2023-10-30T12:09:57.750 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.734+0000 7feb5d80e640 1 librados: calling monclient init 2023-10-30T12:09:57.750 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.734+0000 7feb5d80e640 10 monclient: init 2023-10-30T12:09:57.750 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:57.750 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:09:57.751 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb580adc70 0x7feb5806dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:57.751 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 10 monclient(hunting): picked mon.noname-a con 0x7feb580adc70 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:09:57.751 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb580b4350 0x7feb581de600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:57.751 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 10 monclient(hunting): picked mon.noname-b con 0x7feb580b4350 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:09:57.751 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb5813ebb0 0x7feb581deb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:57.752 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 10 monclient(hunting): picked mon.noname-c con 0x7feb5813ebb0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:09:57.752 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:57.752 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb580b3540 con 0x7feb580adc70 2023-10-30T12:09:57.752 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:57.752 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5806d470 con 0x7feb580b4350 2023-10-30T12:09:57.753 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:57.753 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5806f540 con 0x7feb5813ebb0 2023-10-30T12:09:57.753 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:57.753 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.735+0000 7feb5d80e640 10 monclient(hunting): authenticate will time out at 1388.949748s 2023-10-30T12:09:57.753 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.736+0000 7feb567fc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb580b4350 0x7feb581de600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:57.754 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.736+0000 7feb577fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb5813ebb0 0x7feb581deb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:57.754 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.736+0000 7feb56ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb580adc70 0x7feb5806dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:57.754 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.736+0000 7feb567fc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb580b4350 0x7feb581de600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:52392/0 (socket says 172.21.15.3:52392) 2023-10-30T12:09:57.754 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.736+0000 7feb567fc640 1 -- 172.21.15.3:0/2373872039 learned_addr learned my addr 172.21.15.3:0/2373872039 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:09:57.754 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.736+0000 7feb567fc640 10 monclient(hunting): get_auth_request con 0x7feb580b4350 auth_method 0 2023-10-30T12:09:57.755 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.736+0000 7feb567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:57.755 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.736+0000 7feb567fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:57.755 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.736+0000 7feb567fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:57.755 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb577fe640 10 monclient(hunting): get_auth_request con 0x7feb5813ebb0 auth_method 0 2023-10-30T12:09:57.755 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:57.755 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb577fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:57.756 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb577fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:57.756 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb56ffd640 10 monclient(hunting): get_auth_request con 0x7feb580adc70 auth_method 0 2023-10-30T12:09:57.756 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:57.756 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb56ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:57.756 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb56ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:57.757 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:57.757 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:57.757 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:57.757 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:57.757 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:57.757 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:57.758 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:57.758 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:57.758 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.737+0000 7feb577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:57.758 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.740+0000 7feb567fc640 10 monclient(hunting): handle_auth_done global_id 4216 payload 824 2023-10-30T12:09:57.758 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.740+0000 7feb567fc640 1 -- 172.21.15.3:0/2373872039 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb5813ebb0 msgr2=0x7feb581deb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:57.759 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.740+0000 7feb567fc640 1 --2- 172.21.15.3:0/2373872039 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb5813ebb0 0x7feb581deb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:57.759 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.740+0000 7feb567fc640 1 -- 172.21.15.3:0/2373872039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb580adc70 msgr2=0x7feb5806dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:57.759 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.740+0000 7feb567fc640 1 --2- 172.21.15.3:0/2373872039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb580adc70 0x7feb5806dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:57.759 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.740+0000 7feb567fc640 10 monclient: _finish_hunting 0 2023-10-30T12:09:57.759 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.740+0000 7feb567fc640 1 monclient: found mon.noname-b 2023-10-30T12:09:57.760 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.740+0000 7feb567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:57.760 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.740+0000 7feb567fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:09:57.760 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.740+0000 7feb567fc640 1 -- 172.21.15.3:0/2373872039 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb58141f10 con 0x7feb580b4350 2023-10-30T12:09:57.760 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.742+0000 7feb567fc640 1 --2- 172.21.15.3:0/2373872039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb580b4350 0x7feb581de600 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feb4c00c4a0 tx=0x7feb4c00c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:57.760 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.742+0000 7feb3ffff640 10 client.?.objecter ms_handle_connect 0x7feb580b4350 2023-10-30T12:09:57.761 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.742+0000 7feb3ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:57.763 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.745+0000 7feb3ffff640 1 -- 172.21.15.3:0/2373872039 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c005ce0 con 0x7feb580b4350 2023-10-30T12:09:57.763 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.745+0000 7feb3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:57.764 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.745+0000 7feb3ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:57.764 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.745+0000 7feb3ffff640 10 monclient: dump: 2023-10-30T12:09:57.764 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:57.764 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:57.764 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:57.765 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:57.765 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:57.765 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:57.765 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:57.765 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:57.766 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:57.766 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:57.766 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.745+0000 7feb3ffff640 10 monclient: _finish_auth 0 2023-10-30T12:09:57.766 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.745+0000 7feb3ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:09:57.766 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.745+0000 7feb3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:57.767 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.745+0000 7feb3ffff640 1 -- 172.21.15.3:0/2373872039 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4c00e040 con 0x7feb580b4350 2023-10-30T12:09:57.767 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.745+0000 7feb3ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:57.767 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.745+0000 7feb5d80e640 5 monclient: authenticate success, global_id 4216 2023-10-30T12:09:57.767 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.745+0000 7feb3ffff640 1 -- 172.21.15.3:0/2373872039 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c005e90 con 0x7feb580b4350 2023-10-30T12:09:57.767 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.745+0000 7feb3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:57.767 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.745+0000 7feb3ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:57.768 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.745+0000 7feb3ffff640 10 monclient: dump: 2023-10-30T12:09:57.768 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:09:57.768 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:09:57.768 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:57.768 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:09:57.769 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:57.769 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:09:57.769 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:09:57.769 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:09:57.769 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:09:57.770 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:09:57.770 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.745+0000 7feb557fa640 4 set_mon_vals no callback set 2023-10-30T12:09:57.770 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.746+0000 7feb5d80e640 10 monclient: _renew_subs 2023-10-30T12:09:57.770 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.746+0000 7feb5d80e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:57.770 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.746+0000 7feb5d80e640 1 -- 172.21.15.3:0/2373872039 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb5806d070 con 0x7feb580b4350 2023-10-30T12:09:57.770 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.746+0000 7feb5d80e640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:57.771 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.746+0000 7feb5d80e640 10 monclient: _renew_subs 2023-10-30T12:09:57.771 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.746+0000 7feb5d80e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:57.771 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.746+0000 7feb5d80e640 1 -- 172.21.15.3:0/2373872039 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb581df260 con 0x7feb580b4350 2023-10-30T12:09:57.771 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.747+0000 7feb5d80e640 1 librados: init done 2023-10-30T12:09:57.771 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.748+0000 7feb3dffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:57.772 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.748+0000 7feb3dffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:57.772 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.748+0000 7feb3dffb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:57.772 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.748+0000 7feb3dffb640 1 -- 172.21.15.3:0/2373872039 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb30005040 con 0x7feb580b4350 2023-10-30T12:09:57.773 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.766+0000 7feb3ffff640 1 -- 172.21.15.3:0/2373872039 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7feb4c007270 con 0x7feb580b4350 2023-10-30T12:09:57.773 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.766+0000 7feb3ffff640 1 --2- 172.21.15.3:0/2373872039 >> v2:172.21.15.92:6800/1750155735 conn(0x7feb4406c730 0x7feb4406ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:57.773 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.766+0000 7feb56ffd640 1 --2- 172.21.15.3:0/2373872039 >> v2:172.21.15.92:6800/1750155735 conn(0x7feb4406c730 0x7feb4406ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:57.773 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.766+0000 7feb56ffd640 10 monclient: get_auth_request con 0x7feb4406c730 auth_method 0 2023-10-30T12:09:57.774 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.767+0000 7feb3ffff640 10 client.4216.objecter ms_handle_connect 0x7feb4406c730 2023-10-30T12:09:57.774 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:57.767+0000 7feb56ffd640 1 --2- 172.21.15.3:0/2373872039 >> v2:172.21.15.92:6800/1750155735 conn(0x7feb4406c730 0x7feb4406ebf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feb48007fd0 tx=0x7feb48009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:58.075 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.073+0000 7feb3ffff640 1 -- 172.21.15.3:0/2373872039 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7feb4c057890 con 0x7feb580b4350 2023-10-30T12:09:58.075 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.073+0000 7feb3ffff640 10 client.4216.objecter ms_dispatch 0x7feb58000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T12:09:58.076 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.073+0000 7feb3ffff640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T12:09:58.076 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.073+0000 7feb3ffff640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2023-10-30T12:09:58.076 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.073+0000 7feb3ffff640 20 client.4216.objecter dump_active .. 0 homeless 2023-10-30T12:09:58.076 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.074+0000 7feb3ffff640 1 -- 172.21.15.3:0/2373872039 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feb581df260 con 0x7feb580b4350 2023-10-30T12:09:58.076 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.074+0000 7feb3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:58.077 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.074+0000 7feb3ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:58.456 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.455+0000 7feb3dffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:58.457 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.455+0000 7feb3dffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:58.457 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.455+0000 7feb3dffb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:09:58.457 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.455+0000 7feb3dffb640 1 -- 172.21.15.3:0/2373872039 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb300055c0 con 0x7feb580b4350 2023-10-30T12:09:58.482 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.481+0000 7feb3ffff640 1 -- 172.21.15.3:0/2373872039 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7feb4c05b930 con 0x7feb580b4350 2023-10-30T12:09:58.483 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.481+0000 7feb3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:09:58.483 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.481+0000 7feb3ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:58.483 INFO:teuthology.misc.health.smithi003.stdout: 2023-10-30T12:09:58.483 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"9bae44ae-7379-4ab2-a454-4d550607110c","created":"2023-10-30T12:09:17.680340+0000","modified":"2023-10-30T12:09:24.800567+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:09:24.800567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69005df2-28b6-4bb2-b3e6-79df0d403c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6466ea5c-d74a-4709-a83a-f5d4fe17157e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"69ec5657-9972-4f4b-8012-83886eb47429","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3dad24-4aeb-47a3-ad1d-d13c77cafad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d38d9d7f-b0de-48ab-a850-499c6a0a4364","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b39d2b0a-84ef-4909-bac7-47f146305a79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0a9d81ac-5a14-4fef-877a-982594787dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:09:58.486 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 10 librados: watch_flush enter 2023-10-30T12:09:58.486 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 10 librados: watch_flush exit 2023-10-30T12:09:58.486 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 20 client.4216.objecter shutdown clearing up homeless session... 2023-10-30T12:09:58.487 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 10 client.4216.objecter successfully canceled tick 2023-10-30T12:09:58.487 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 1 -- 172.21.15.3:0/2373872039 >> v2:172.21.15.92:6800/1750155735 conn(0x7feb4406c730 msgr2=0x7feb4406ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:58.487 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 1 --2- 172.21.15.3:0/2373872039 >> v2:172.21.15.92:6800/1750155735 conn(0x7feb4406c730 0x7feb4406ebf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feb48007fd0 tx=0x7feb48009040 comp rx=0 tx=0).stop 2023-10-30T12:09:58.487 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 10 monclient: shutdown 2023-10-30T12:09:58.488 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:58.488 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 1 -- 172.21.15.3:0/2373872039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb580b4350 msgr2=0x7feb581de600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:58.488 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 1 --2- 172.21.15.3:0/2373872039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb580b4350 0x7feb581de600 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feb4c00c4a0 tx=0x7feb4c00c960 comp rx=0 tx=0).stop 2023-10-30T12:09:58.488 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 1 -- 172.21.15.3:0/2373872039 shutdown_connections 2023-10-30T12:09:58.488 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 1 --2- 172.21.15.3:0/2373872039 >> v2:172.21.15.92:6800/1750155735 conn(0x7feb4406c730 0x7feb4406ebf0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:58.489 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 1 --2- 172.21.15.3:0/2373872039 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb5813ebb0 0x7feb581deb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:58.489 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 1 --2- 172.21.15.3:0/2373872039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb580b4350 0x7feb581de600 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:58.489 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 1 --2- 172.21.15.3:0/2373872039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb580adc70 0x7feb5806dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:58.489 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 1 -- 172.21.15.3:0/2373872039 >> 172.21.15.3:0/2373872039 conn(0x7feb58060bf0 msgr2=0x7feb58135950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:58.489 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 1 -- 172.21.15.3:0/2373872039 shutdown_connections 2023-10-30T12:09:58.490 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 1 -- 172.21.15.3:0/2373872039 wait complete. 2023-10-30T12:09:58.490 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:09:58.484+0000 7feb3dffb640 1 librados: shutdown 2023-10-30T12:09:58.500 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T12:10:00.808 INFO:tasks.ceph.osd.4.smithi092.stderr:2023-10-30T12:10:00.803+0000 5dc2340 -1 osd.4 0 log_to_monitors true 2023-10-30T12:10:00.947 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-10-30T12:10:00.889+0000 5dc2340 -1 osd.6 0 log_to_monitors true 2023-10-30T12:10:01.124 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-10-30T12:10:01.120+0000 5dc2340 -1 osd.5 0 log_to_monitors true 2023-10-30T12:10:01.322 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-10-30T12:10:01.316+0000 5dc2340 -1 osd.7 0 log_to_monitors true 2023-10-30T12:10:01.991 INFO:tasks.ceph.osd.1.smithi003.stderr:2023-10-30T12:10:01.987+0000 5dc2340 -1 osd.1 0 log_to_monitors true 2023-10-30T12:10:02.196 INFO:tasks.ceph.osd.0.smithi003.stderr:2023-10-30T12:10:02.190+0000 5dc2340 -1 osd.0 0 log_to_monitors true 2023-10-30T12:10:02.421 INFO:tasks.ceph.osd.3.smithi003.stderr:2023-10-30T12:10:02.417+0000 5dc2340 -1 osd.3 0 log_to_monitors true 2023-10-30T12:10:02.687 INFO:tasks.ceph.osd.2.smithi003.stderr:2023-10-30T12:10:02.683+0000 5dc2340 -1 osd.2 0 log_to_monitors true 2023-10-30T12:10:03.974 INFO:tasks.ceph.osd.4.smithi092.stderr:2023-10-30T12:10:03.969+0000 b47e640 -1 osd.4 0 waiting for initial osdmap 2023-10-30T12:10:03.991 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-10-30T12:10:03.983+0000 b47e640 -1 osd.5 0 waiting for initial osdmap 2023-10-30T12:10:04.046 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-10-30T12:10:04.033+0000 b47e640 -1 osd.6 0 waiting for initial osdmap 2023-10-30T12:10:04.354 INFO:tasks.ceph.osd.4.smithi092.stderr:2023-10-30T12:10:04.340+0000 10a5b640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:10:04.416 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-10-30T12:10:04.397+0000 10a5b640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:10:04.426 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-10-30T12:10:04.412+0000 10a5b640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:10:05.066 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-10-30T12:10:05.060+0000 bc94640 -1 osd.7 0 waiting for initial osdmap 2023-10-30T12:10:05.167 INFO:tasks.ceph.osd.1.smithi003.stderr:2023-10-30T12:10:05.161+0000 b47e640 -1 osd.1 0 waiting for initial osdmap 2023-10-30T12:10:05.306 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:10:05.400 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.398+0000 7fbace021640 10 monclient: get_monmap_and_config 2023-10-30T12:10:05.400 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.398+0000 7fbace021640 10 monclient: build_initial_monmap 2023-10-30T12:10:05.400 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.398+0000 7fbace021640 10 monclient: monmap: 2023-10-30T12:10:05.400 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2023-10-30T12:10:05.401 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:05.401 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:10:05.399714+0000 2023-10-30T12:10:05.401 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:10:05.399714+0000 2023-10-30T12:10:05.401 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:05.401 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:05.402 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:05.402 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:05.402 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:05.402 INFO:teuthology.misc.health.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:05.402 INFO:teuthology.misc.health.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:05.403 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:10:05.405 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.399+0000 7fbace021640 1 Processor -- start 2023-10-30T12:10:05.405 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.399+0000 7fbace021640 1 -- start start 2023-10-30T12:10:05.405 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.399+0000 7fbace021640 10 monclient: init 2023-10-30T12:10:05.406 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.400+0000 7fbace021640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:05.406 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.400+0000 7fbace021640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-10-30T12:10:05.406 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.400+0000 7fbace021640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbac80b4350 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:05.407 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.400+0000 7fbace021640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbac80adc70 0x7fbac80b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:05.407 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.400+0000 7fbace021640 10 monclient(hunting): picked mon.noname-b con 0x7fbac80adc70 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:05.408 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.400+0000 7fbace021640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbac80b0d50 0x7fbac813ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:05.408 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.400+0000 7fbace021640 10 monclient(hunting): picked mon.noname-a con 0x7fbac80b0d50 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:05.408 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.400+0000 7fbace021640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac80ae950 con 0x7fbac80b4350 2023-10-30T12:10:05.409 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.400+0000 7fbace021640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:05.409 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.400+0000 7fbace021640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac8067a10 con 0x7fbac80b0d50 2023-10-30T12:10:05.409 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.400+0000 7fbace021640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:05.410 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.400+0000 7fbace021640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac80b4730 con 0x7fbac80adc70 2023-10-30T12:10:05.410 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.400+0000 7fbace021640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:05.410 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.400+0000 7fbace021640 10 monclient(hunting): authenticate will time out at 1396.614362s 2023-10-30T12:10:05.411 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.401+0000 7fbac77fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbac80b4350 0x7fbac80aee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:37646/0 (socket says 172.21.15.3:37646) 2023-10-30T12:10:05.411 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.401+0000 7fbabeffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbac80adc70 0x7fbac80b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:05.412 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.401+0000 7fbac77fe640 1 -- 172.21.15.3:0/3494614377 learned_addr learned my addr 172.21.15.3:0/3494614377 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:05.412 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.401+0000 7fbac7fff640 1 --2- 172.21.15.3:0/3494614377 >> v2:172.21.15.3:3300/0 conn(0x7fbac80b0d50 0x7fbac813ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:05.412 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.402+0000 7fbac7fff640 10 monclient(hunting): get_auth_request con 0x7fbac80b0d50 auth_method 0 2023-10-30T12:10:05.413 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.402+0000 7fbac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:05.413 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.402+0000 7fbac7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:05.413 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.402+0000 7fbac7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:05.414 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.402+0000 7fbabeffd640 10 monclient(hunting): get_auth_request con 0x7fbac80adc70 auth_method 0 2023-10-30T12:10:05.414 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.402+0000 7fbabeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:05.414 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.402+0000 7fbabeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:05.414 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.402+0000 7fbabeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:05.417 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.404+0000 7fbac7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:05.417 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.404+0000 7fbac7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:05.417 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.404+0000 7fbac7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:05.418 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.405+0000 7fbac6ffd640 1 -- 172.21.15.3:0/3494614377 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719451867 0 0) 0x7fbab8001110 con 0x7fbac80b4350 2023-10-30T12:10:05.418 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.405+0000 7fbac6ffd640 10 monclient(hunting): _negotiate 2023-10-30T12:10:05.418 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.405+0000 7fbac6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:05.418 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.405+0000 7fbac6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:05.418 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.405+0000 7fbac6ffd640 10 monclient(hunting): my global_id is 4263 2023-10-30T12:10:05.419 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.405+0000 7fbac6ffd640 1 -- 172.21.15.3:0/3494614377 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab4003520 con 0x7fbac80b4350 2023-10-30T12:10:05.419 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.405+0000 7fbabeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:05.419 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.405+0000 7fbabeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:05.419 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.405+0000 7fbabeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:05.420 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.406+0000 7fbac7fff640 10 monclient(hunting): handle_auth_done global_id 4266 payload 274 2023-10-30T12:10:05.420 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.406+0000 7fbac7fff640 1 -- 172.21.15.3:0/3494614377 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbac80adc70 msgr2=0x7fbac80b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:05.420 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.406+0000 7fbac7fff640 1 --2- 172.21.15.3:0/3494614377 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbac80adc70 0x7fbac80b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:05.420 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.406+0000 7fbac7fff640 1 -- 172.21.15.3:0/3494614377 >> v1:172.21.15.3:6789/0 conn(0x7fbac80b4350 legacy=0x7fbac80aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:05.420 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.406+0000 7fbac7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:05.421 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.406+0000 7fbac7fff640 1 monclient: found mon.noname-a 2023-10-30T12:10:05.421 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.406+0000 7fbac7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:05.421 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.406+0000 7fbac7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:05.421 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.406+0000 7fbac7fff640 1 -- 172.21.15.3:0/3494614377 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac80b3990 con 0x7fbac80b0d50 2023-10-30T12:10:05.421 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.409+0000 7fbac7fff640 1 --2- 172.21.15.3:0/3494614377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbac80b0d50 0x7fbac813ebb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009d90 tx=0x7fbac002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45202a7af10a860f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:05.422 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.414+0000 7fbac6ffd640 1 -- 172.21.15.3:0/3494614377 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac002cbc0 con 0x7fbac80b0d50 2023-10-30T12:10:05.422 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:05.422 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbac6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:05.422 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbac6ffd640 10 monclient: dump: 2023-10-30T12:10:05.423 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:10:05.423 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:05.423 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:05.423 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:05.423 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:05.424 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:05.424 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:05.424 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:05.425 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:05.425 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:10:05.425 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbac6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:10:05.426 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbac6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:05.426 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:05.426 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbac6ffd640 1 -- 172.21.15.3:0/3494614377 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac002cd30 con 0x7fbac80b0d50 2023-10-30T12:10:05.427 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbace021640 5 monclient: authenticate success, global_id 4266 2023-10-30T12:10:05.427 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbace021640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:05.428 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbac6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:05.428 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbace021640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:05.428 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbac6ffd640 1 -- 172.21.15.3:0/3494614377 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac0034660 con 0x7fbac80b0d50 2023-10-30T12:10:05.429 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:05.429 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbace021640 4 set_mon_vals no callback set 2023-10-30T12:10:05.429 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbac6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:05.430 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbac6ffd640 10 monclient: dump: 2023-10-30T12:10:05.430 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:10:05.430 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:05.431 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:05.431 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:05.431 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:05.432 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:05.432 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:05.432 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:05.433 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:05.433 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:10:05.433 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbace021640 10 monclient: shutdown 2023-10-30T12:10:05.434 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbace021640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:05.434 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbace021640 1 -- 172.21.15.3:0/3494614377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbac80b0d50 msgr2=0x7fbac813ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:05.434 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbace021640 1 --2- 172.21.15.3:0/3494614377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbac80b0d50 0x7fbac813ebb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009d90 tx=0x7fbac002be10 comp rx=0 tx=0).stop 2023-10-30T12:10:05.434 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbace021640 1 -- 172.21.15.3:0/3494614377 shutdown_connections 2023-10-30T12:10:05.435 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbace021640 1 --2- 172.21.15.3:0/3494614377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbac80b0d50 0x7fbac813ebb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:05.435 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbace021640 1 --2- 172.21.15.3:0/3494614377 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbac80adc70 0x7fbac80b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:05.435 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbace021640 1 -- 172.21.15.3:0/3494614377 >> 172.21.15.3:0/3494614377 conn(0x7fbac8060bf0 msgr2=0x7fbac8061000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:05.435 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbace021640 1 -- 172.21.15.3:0/3494614377 shutdown_connections 2023-10-30T12:10:05.435 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.415+0000 7fbace021640 1 -- 172.21.15.3:0/3494614377 wait complete. 2023-10-30T12:10:05.440 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.416+0000 7fbace021640 10 monclient: build_initial_monmap 2023-10-30T12:10:05.440 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.416+0000 7fbace021640 10 monclient: monmap: 2023-10-30T12:10:05.440 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2023-10-30T12:10:05.441 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:05.441 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:05.441 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2023-10-30T12:10:05.441 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:05.441 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:05.442 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:05.442 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:05.442 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:05.442 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:10:05.442 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.416+0000 7fbace021640 1 librados: starting msgr at 2023-10-30T12:10:05.443 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.416+0000 7fbace021640 1 librados: starting objecter 2023-10-30T12:10:05.443 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.416+0000 7fbace021640 1 Processor -- start 2023-10-30T12:10:05.443 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 1 -- start start 2023-10-30T12:10:05.443 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 1 librados: setting wanted keys 2023-10-30T12:10:05.443 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 1 librados: calling monclient init 2023-10-30T12:10:05.443 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 10 monclient: init 2023-10-30T12:10:05.444 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:05.444 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:10:05.444 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbac80adc70 0x7fbac811f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:05.444 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 10 monclient(hunting): picked mon.noname-c con 0x7fbac80adc70 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:05.444 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbac80b0d50 0x7fbac80ad030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:05.445 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 10 monclient(hunting): picked mon.noname-a con 0x7fbac80b0d50 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:05.445 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbac80b4350 0x7fbac81da290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:05.445 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 10 monclient(hunting): picked mon.noname-b con 0x7fbac80b4350 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:05.445 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:05.445 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac80aff50 con 0x7fbac80b0d50 2023-10-30T12:10:05.446 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:05.446 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac81d9e00 con 0x7fbac80b4350 2023-10-30T12:10:05.446 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:05.446 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac811fcd0 con 0x7fbac80adc70 2023-10-30T12:10:05.446 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:05.447 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.417+0000 7fbace021640 10 monclient(hunting): authenticate will time out at 1396.631896s 2023-10-30T12:10:05.447 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbac7fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbac80b4350 0x7fbac81da290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:05.447 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbac77fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbac80adc70 0x7fbac811f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:05.447 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbabeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbac80b0d50 0x7fbac80ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:05.447 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbac7fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbac80b4350 0x7fbac81da290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:52638/0 (socket says 172.21.15.3:52638) 2023-10-30T12:10:05.448 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbac7fff640 1 -- 172.21.15.3:0/2128678269 learned_addr learned my addr 172.21.15.3:0/2128678269 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:05.448 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbac7fff640 10 monclient(hunting): get_auth_request con 0x7fbac80b4350 auth_method 0 2023-10-30T12:10:05.448 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:05.448 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbac7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:05.448 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbac7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:05.449 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbabeffd640 10 monclient(hunting): get_auth_request con 0x7fbac80b0d50 auth_method 0 2023-10-30T12:10:05.449 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbabeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:05.449 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbabeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:05.449 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbabeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:05.449 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbac77fe640 10 monclient(hunting): get_auth_request con 0x7fbac80adc70 auth_method 0 2023-10-30T12:10:05.450 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbac77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:05.450 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbac77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:05.450 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.418+0000 7fbac77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:05.451 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.419+0000 7fbac7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:05.451 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.419+0000 7fbac7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:05.451 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.419+0000 7fbac7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:05.452 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.419+0000 7fbabeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:05.452 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.419+0000 7fbabeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:05.452 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.419+0000 7fbabeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:05.453 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.419+0000 7fbac77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:05.453 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.419+0000 7fbac77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:05.453 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.419+0000 7fbac77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:05.454 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.422+0000 7fbac7fff640 10 monclient(hunting): handle_auth_done global_id 4240 payload 824 2023-10-30T12:10:05.454 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.422+0000 7fbac7fff640 1 -- 172.21.15.3:0/2128678269 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbac80adc70 msgr2=0x7fbac811f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:05.455 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.422+0000 7fbac7fff640 1 --2- 172.21.15.3:0/2128678269 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbac80adc70 0x7fbac811f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:05.455 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.422+0000 7fbac7fff640 1 -- 172.21.15.3:0/2128678269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbac80b0d50 msgr2=0x7fbac80ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:05.455 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.422+0000 7fbac7fff640 1 --2- 172.21.15.3:0/2128678269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbac80b0d50 0x7fbac80ad030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:05.456 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.422+0000 7fbac7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:05.456 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.422+0000 7fbac7fff640 1 monclient: found mon.noname-b 2023-10-30T12:10:05.456 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.422+0000 7fbac7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:05.457 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.422+0000 7fbac7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:05.457 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.422+0000 7fbac7fff640 1 -- 172.21.15.3:0/2128678269 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac80ad570 con 0x7fbac80b4350 2023-10-30T12:10:05.457 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.424+0000 7fbac7fff640 1 --2- 172.21.15.3:0/2128678269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbac80b4350 0x7fbac81da290 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009eb0 tx=0x7fbac0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:05.458 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.424+0000 7fbac4ff9640 10 client.?.objecter ms_handle_connect 0x7fbac80b4350 2023-10-30T12:10:05.458 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.424+0000 7fbac4ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:05.459 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbac4ff9640 1 -- 172.21.15.3:0/2128678269 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac0040650 con 0x7fbac80b4350 2023-10-30T12:10:05.459 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbac4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:05.459 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbac4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:05.460 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbac4ff9640 10 monclient: dump: 2023-10-30T12:10:05.460 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:10:05.460 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:05.461 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:05.461 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:05.461 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:05.462 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:05.462 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:05.462 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:05.463 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:05.463 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:10:05.463 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbac4ff9640 10 monclient: _finish_auth 0 2023-10-30T12:10:05.464 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbac4ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:10:05.464 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbac4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:05.465 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbac4ff9640 1 -- 172.21.15.3:0/2128678269 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac00407c0 con 0x7fbac80b4350 2023-10-30T12:10:05.465 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbace021640 5 monclient: authenticate success, global_id 4240 2023-10-30T12:10:05.465 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbace021640 10 monclient: _renew_subs 2023-10-30T12:10:05.466 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbace021640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:05.466 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbace021640 1 -- 172.21.15.3:0/2128678269 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac80ac3f0 con 0x7fbac80b4350 2023-10-30T12:10:05.470 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbac4ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:05.471 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbac4ff9640 1 -- 172.21.15.3:0/2128678269 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac0040b80 con 0x7fbac80b4350 2023-10-30T12:10:05.471 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbac4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:05.471 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbac4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:05.472 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbac4ff9640 10 monclient: dump: 2023-10-30T12:10:05.472 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:10:05.472 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:05.473 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:05.473 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:05.473 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:05.474 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:05.474 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:05.475 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:05.475 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:05.475 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:10:05.475 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbac67fc640 4 set_mon_vals no callback set 2023-10-30T12:10:05.476 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.427+0000 7fbace021640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:05.476 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.428+0000 7fbace021640 10 monclient: _renew_subs 2023-10-30T12:10:05.476 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.428+0000 7fbace021640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:05.477 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.428+0000 7fbace021640 1 -- 172.21.15.3:0/2128678269 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac80aa700 con 0x7fbac80b4350 2023-10-30T12:10:05.477 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.428+0000 7fbace021640 1 librados: init done 2023-10-30T12:10:05.477 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.428+0000 7fbace021640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:05.478 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.428+0000 7fbace021640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:05.478 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.428+0000 7fbace021640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:05.478 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.428+0000 7fbace021640 1 -- 172.21.15.3:0/2128678269 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac81d9fb0 con 0x7fbac80b4350 2023-10-30T12:10:05.480 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.447+0000 7fbac4ff9640 1 -- 172.21.15.3:0/2128678269 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7fbac002cbd0 con 0x7fbac80b4350 2023-10-30T12:10:05.481 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.447+0000 7fbac4ff9640 1 --2- 172.21.15.3:0/2128678269 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbaa806c780 0x7fbaa806ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:05.481 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.447+0000 7fbac77fe640 1 --2- 172.21.15.3:0/2128678269 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbaa806c780 0x7fbaa806ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:05.482 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.447+0000 7fbac77fe640 10 monclient: get_auth_request con 0x7fbaa806c780 auth_method 0 2023-10-30T12:10:05.482 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.448+0000 7fbac77fe640 1 --2- 172.21.15.3:0/2128678269 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbaa806c780 0x7fbaa806ec40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbab8001380 tx=0x7fbab8007b90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:05.482 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.448+0000 7fbac4ff9640 10 client.4240.objecter ms_handle_connect 0x7fbaa806c780 2023-10-30T12:10:05.483 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.450+0000 7fbac4ff9640 1 -- 172.21.15.3:0/2128678269 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2603+0+0 (secure 0 0 0) 0x7fbac007a320 con 0x7fbac80b4350 2023-10-30T12:10:05.483 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.450+0000 7fbac4ff9640 10 client.4240.objecter ms_dispatch 0x7fbac8000b90 osd_map(13..13 src has 1..13) v4 2023-10-30T12:10:05.483 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.450+0000 7fbac4ff9640 3 client.4240.objecter handle_osd_map got epochs [13,13] > 0 2023-10-30T12:10:05.483 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.450+0000 7fbac4ff9640 3 client.4240.objecter handle_osd_map decoding full epoch 13 2023-10-30T12:10:05.483 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.450+0000 7fbac4ff9640 20 client.4240.objecter dump_active .. 0 homeless 2023-10-30T12:10:05.497 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-10-30T12:10:05.481+0000 10a5b640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:10:05.592 INFO:tasks.ceph.osd.1.smithi003.stderr:2023-10-30T12:10:05.578+0000 1125c640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:10:05.707 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.706+0000 7fbac4ff9640 1 -- 172.21.15.3:0/2128678269 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbac007e3c0 con 0x7fbac80b4350 2023-10-30T12:10:05.708 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.706+0000 7fbac4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:05.708 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.706+0000 7fbac4ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:05.905 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.903+0000 7fbace021640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:10:05.905 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.903+0000 7fbace021640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:10:05.906 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.903+0000 7fbace021640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:05.906 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.903+0000 7fbace021640 1 -- 172.21.15.3:0/2128678269 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbac80ad570 con 0x7fbac80b4350 2023-10-30T12:10:05.921 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.919+0000 7fbac4ff9640 1 -- 172.21.15.3:0/2128678269 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+7782 (secure 0 0 0) 0x7fbac0082570 con 0x7fbac80b4350 2023-10-30T12:10:05.921 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.919+0000 7fbac4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:10:05.922 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.919+0000 7fbac4ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:05.922 INFO:teuthology.misc.health.smithi003.stdout: 2023-10-30T12:10:05.922 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":13,"fsid":"9bae44ae-7379-4ab2-a454-4d550607110c","created":"2023-10-30T12:09:17.680340+0000","modified":"2023-10-30T12:10:04.665889+0000","last_up_change":"2023-10-30T12:10:04.665889+0000","last_in_change":"2023-10-30T12:09:24.800567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69005df2-28b6-4bb2-b3e6-79df0d403c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6466ea5c-d74a-4709-a83a-f5d4fe17157e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"69ec5657-9972-4f4b-8012-83886eb47429","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3dad24-4aeb-47a3-ad1d-d13c77cafad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6801","nonce":4229938171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4229938171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4229938171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6803","nonce":4229938171}]},"public_addr":"172.21.15.92:6801/4229938171","cluster_addr":"172.21.15.92:6802/4229938171","heartbeat_back_addr":"172.21.15.92:6804/4229938171","heartbeat_front_addr":"172.21.15.92:6803/4229938171","state":["exists","up"]},{"osd":5,"uuid":"d38d9d7f-b0de-48ab-a850-499c6a0a4364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6805","nonce":3208162957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3208162957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3208162957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6807","nonce":3208162957}]},"public_addr":"172.21.15.92:6805/3208162957","cluster_addr":"172.21.15.92:6806/3208162957","heartbeat_back_addr":"172.21.15.92:6808/3208162957","heartbeat_front_addr":"172.21.15.92:6807/3208162957","state":["exists","up"]},{"osd":6,"uuid":"b39d2b0a-84ef-4909-bac7-47f146305a79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":3275842310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3275842310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3275842310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6811","nonce":3275842310}]},"public_addr":"172.21.15.92:6809/3275842310","cluster_addr":"172.21.15.92:6810/3275842310","heartbeat_back_addr":"172.21.15.92:6812/3275842310","heartbeat_front_addr":"172.21.15.92:6811/3275842310","state":["exists","up"]},{"osd":7,"uuid":"0a9d81ac-5a14-4fef-877a-982594787dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:10:05.924 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 10 librados: watch_flush enter 2023-10-30T12:10:05.924 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 10 librados: watch_flush exit 2023-10-30T12:10:05.924 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 20 client.4240.objecter shutdown clearing up homeless session... 2023-10-30T12:10:05.925 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 10 client.4240.objecter successfully canceled tick 2023-10-30T12:10:05.925 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 1 -- 172.21.15.3:0/2128678269 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbaa806c780 msgr2=0x7fbaa806ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:05.925 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 1 --2- 172.21.15.3:0/2128678269 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbaa806c780 0x7fbaa806ec40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbab8001380 tx=0x7fbab8007b90 comp rx=0 tx=0).stop 2023-10-30T12:10:05.925 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 10 monclient: shutdown 2023-10-30T12:10:05.926 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:05.926 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 1 -- 172.21.15.3:0/2128678269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbac80b4350 msgr2=0x7fbac81da290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:05.926 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 1 --2- 172.21.15.3:0/2128678269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbac80b4350 0x7fbac81da290 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009eb0 tx=0x7fbac0004a40 comp rx=0 tx=0).stop 2023-10-30T12:10:05.926 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 1 -- 172.21.15.3:0/2128678269 shutdown_connections 2023-10-30T12:10:05.926 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 1 --2- 172.21.15.3:0/2128678269 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbaa806c780 0x7fbaa806ec40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:05.927 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 1 --2- 172.21.15.3:0/2128678269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbac80b4350 0x7fbac81da290 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:05.927 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 1 --2- 172.21.15.3:0/2128678269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbac80b0d50 0x7fbac80ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:05.927 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 1 --2- 172.21.15.3:0/2128678269 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbac80adc70 0x7fbac811f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:05.927 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 1 -- 172.21.15.3:0/2128678269 >> 172.21.15.3:0/2128678269 conn(0x7fbac8060bf0 msgr2=0x7fbac80b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:05.927 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 1 -- 172.21.15.3:0/2128678269 shutdown_connections 2023-10-30T12:10:05.928 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 1 -- 172.21.15.3:0/2128678269 wait complete. 2023-10-30T12:10:05.928 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:05.922+0000 7fbace021640 1 librados: shutdown 2023-10-30T12:10:05.934 DEBUG:teuthology.misc:3 of 8 OSDs are up 2023-10-30T12:10:06.294 INFO:tasks.ceph.osd.0.smithi003.stderr:2023-10-30T12:10:06.290+0000 bc94640 -1 osd.0 0 waiting for initial osdmap 2023-10-30T12:10:06.295 INFO:tasks.ceph.osd.2.smithi003.stderr:2023-10-30T12:10:06.293+0000 bc94640 -1 osd.2 0 waiting for initial osdmap 2023-10-30T12:10:06.316 INFO:tasks.ceph.osd.3.smithi003.stderr:2023-10-30T12:10:06.310+0000 bc94640 -1 osd.3 0 waiting for initial osdmap 2023-10-30T12:10:06.700 INFO:tasks.ceph.osd.0.smithi003.stderr:2023-10-30T12:10:06.686+0000 1125c640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:10:06.736 INFO:tasks.ceph.osd.3.smithi003.stderr:2023-10-30T12:10:06.722+0000 11a5d640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:10:06.762 INFO:tasks.ceph.osd.2.smithi003.stderr:2023-10-30T12:10:06.748+0000 1125c640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:10:12.741 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:10:12.833 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.832+0000 7fe917c7c640 10 monclient: get_monmap_and_config 2023-10-30T12:10:12.833 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.832+0000 7fe917c7c640 10 monclient: build_initial_monmap 2023-10-30T12:10:12.834 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.832+0000 7fe917c7c640 10 monclient: monmap: 2023-10-30T12:10:12.834 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2023-10-30T12:10:12.834 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:12.834 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:10:12.833120+0000 2023-10-30T12:10:12.834 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:10:12.833120+0000 2023-10-30T12:10:12.835 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:12.835 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:12.835 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:12.835 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:12.835 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:12.836 INFO:teuthology.misc.health.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:12.836 INFO:teuthology.misc.health.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:12.836 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:10:12.838 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.832+0000 7fe917c7c640 1 Processor -- start 2023-10-30T12:10:12.839 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.832+0000 7fe917c7c640 1 -- start start 2023-10-30T12:10:12.839 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.832+0000 7fe917c7c640 10 monclient: init 2023-10-30T12:10:12.839 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.833+0000 7fe917c7c640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:12.839 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.833+0000 7fe917c7c640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2023-10-30T12:10:12.839 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.833+0000 7fe917c7c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe9100b4350 0x7fe9100aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:12.840 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.833+0000 7fe917c7c640 10 monclient(hunting): picked mon.noname-c con 0x7fe9100b4350 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:12.840 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.833+0000 7fe917c7c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe9100adc70 0x7fe9100b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:12.840 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.833+0000 7fe917c7c640 10 monclient(hunting): picked mon.noname-a con 0x7fe9100adc70 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:12.840 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.833+0000 7fe917c7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9100b0d50 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:12.841 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.833+0000 7fe917c7c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9100af3b0 con 0x7fe9100b0d50 2023-10-30T12:10:12.841 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.833+0000 7fe917c7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:12.841 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.833+0000 7fe917c7c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe910067a10 con 0x7fe9100adc70 2023-10-30T12:10:12.841 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.833+0000 7fe917c7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:12.841 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.833+0000 7fe917c7c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9100b4730 con 0x7fe9100b4350 2023-10-30T12:10:12.841 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.833+0000 7fe917c7c640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:12.842 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.833+0000 7fe917c7c640 10 monclient(hunting): authenticate will time out at 1404.047717s 2023-10-30T12:10:12.842 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.834+0000 7fe915217640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe9100adc70 0x7fe9100b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:12.842 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.834+0000 7fe915a18640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe9100b4350 0x7fe9100aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:12.842 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.834+0000 7fe915217640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe9100adc70 0x7fe9100b0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42128/0 (socket says 172.21.15.3:42128) 2023-10-30T12:10:12.842 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.834+0000 7fe915217640 1 -- 172.21.15.3:0/3832997383 learned_addr learned my addr 172.21.15.3:0/3832997383 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:12.843 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.834+0000 7fe915217640 10 monclient(hunting): get_auth_request con 0x7fe9100adc70 auth_method 0 2023-10-30T12:10:12.843 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.834+0000 7fe915217640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:12.843 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.834+0000 7fe915217640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:12.843 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.834+0000 7fe915217640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:12.843 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.834+0000 7fe915a18640 10 monclient(hunting): get_auth_request con 0x7fe9100b4350 auth_method 0 2023-10-30T12:10:12.843 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.834+0000 7fe915a18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:12.844 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.834+0000 7fe915a18640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:12.844 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.834+0000 7fe915a18640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:12.844 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.835+0000 7fe915a18640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:12.844 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.835+0000 7fe915a18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:12.844 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.835+0000 7fe915a18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:12.845 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.835+0000 7fe915217640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:12.845 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.835+0000 7fe915217640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:12.845 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.835+0000 7fe915217640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:12.845 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.836+0000 7fe915a18640 10 monclient(hunting): handle_auth_done global_id 4249 payload 274 2023-10-30T12:10:12.845 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.836+0000 7fe915a18640 1 -- 172.21.15.3:0/3832997383 >> v2:172.21.15.3:3300/0 conn(0x7fe9100adc70 msgr2=0x7fe9100b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:12.845 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.836+0000 7fe915a18640 1 --2- 172.21.15.3:0/3832997383 >> v2:172.21.15.3:3300/0 conn(0x7fe9100adc70 0x7fe9100b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:12.846 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.836+0000 7fe915a18640 1 -- 172.21.15.3:0/3832997383 >> v1:172.21.15.3:6789/0 conn(0x7fe9100b0d50 legacy=0x7fe91013ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:12.846 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.836+0000 7fe915a18640 10 monclient: _finish_hunting 0 2023-10-30T12:10:12.846 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.836+0000 7fe915a18640 1 monclient: found mon.noname-c 2023-10-30T12:10:12.846 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.836+0000 7fe915a18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:12.846 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.836+0000 7fe915a18640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:10:12.847 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.836+0000 7fe915a18640 1 -- 172.21.15.3:0/3832997383 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9100ae950 con 0x7fe9100b4350 2023-10-30T12:10:12.851 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.838+0000 7fe915a18640 1 --2- 172.21.15.3:0/3832997383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9100b4350 0x7fe9100aee70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe9100b3340 tx=0x7fe90002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93c32de696ea9b3d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:12.851 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe914a16640 1 -- 172.21.15.3:0/3832997383 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9000315a0 con 0x7fe9100b4350 2023-10-30T12:10:12.851 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe914a16640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:12.852 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe914a16640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:12.852 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe914a16640 10 monclient: dump: 2023-10-30T12:10:12.852 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:10:12.852 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:12.852 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:12.853 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:12.853 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:12.853 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:12.853 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:12.853 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:12.853 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:12.854 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:10:12.854 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe914a16640 10 monclient: _finish_auth 0 2023-10-30T12:10:12.854 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe914a16640 10 monclient: _check_auth_tickets 2023-10-30T12:10:12.854 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe914a16640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:12.854 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe917c7c640 5 monclient: authenticate success, global_id 4249 2023-10-30T12:10:12.855 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe917c7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:12.855 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe914a16640 1 -- 172.21.15.3:0/3832997383 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe900031710 con 0x7fe9100b4350 2023-10-30T12:10:12.855 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe914a16640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:12.855 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe917c7c640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:12.855 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe914a16640 1 -- 172.21.15.3:0/3832997383 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe900031ab0 con 0x7fe9100b4350 2023-10-30T12:10:12.856 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe917c7c640 4 set_mon_vals no callback set 2023-10-30T12:10:12.856 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe914a16640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:12.856 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe914a16640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:12.856 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe914a16640 10 monclient: dump: 2023-10-30T12:10:12.856 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:10:12.856 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:12.857 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:12.857 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:12.857 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:12.857 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:12.857 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:12.858 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:12.858 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:12.858 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:10:12.858 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe917c7c640 10 monclient: shutdown 2023-10-30T12:10:12.858 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe917c7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:12.859 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe917c7c640 1 -- 172.21.15.3:0/3832997383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9100b4350 msgr2=0x7fe9100aee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:12.859 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe917c7c640 1 --2- 172.21.15.3:0/3832997383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9100b4350 0x7fe9100aee70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe9100b3340 tx=0x7fe90002b840 comp rx=0 tx=0).stop 2023-10-30T12:10:12.859 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe917c7c640 1 -- 172.21.15.3:0/3832997383 shutdown_connections 2023-10-30T12:10:12.859 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe917c7c640 1 --2- 172.21.15.3:0/3832997383 >> v2:172.21.15.3:3300/0 conn(0x7fe9100adc70 0x7fe9100b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:12.859 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe917c7c640 1 --2- 172.21.15.3:0/3832997383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9100b4350 0x7fe9100aee70 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:12.860 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.842+0000 7fe917c7c640 1 -- 172.21.15.3:0/3832997383 >> 172.21.15.3:0/3832997383 conn(0x7fe910060bf0 msgr2=0x7fe910061000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:12.860 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.843+0000 7fe917c7c640 1 -- 172.21.15.3:0/3832997383 shutdown_connections 2023-10-30T12:10:12.860 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.843+0000 7fe917c7c640 1 -- 172.21.15.3:0/3832997383 wait complete. 2023-10-30T12:10:12.860 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.843+0000 7fe917c7c640 10 monclient: build_initial_monmap 2023-10-30T12:10:12.860 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.843+0000 7fe917c7c640 10 monclient: monmap: 2023-10-30T12:10:12.861 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2023-10-30T12:10:12.861 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:12.861 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:12.861 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2023-10-30T12:10:12.861 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:12.861 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:12.862 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:12.862 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:12.862 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:12.862 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:10:12.862 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.843+0000 7fe917c7c640 1 librados: starting msgr at 2023-10-30T12:10:12.863 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.843+0000 7fe917c7c640 1 librados: starting objecter 2023-10-30T12:10:12.863 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 1 Processor -- start 2023-10-30T12:10:12.863 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 1 -- start start 2023-10-30T12:10:12.863 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 1 librados: setting wanted keys 2023-10-30T12:10:12.863 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 1 librados: calling monclient init 2023-10-30T12:10:12.864 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 10 monclient: init 2023-10-30T12:10:12.864 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:12.864 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:10:12.864 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9100adc70 0x7fe9100ad210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:12.864 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 10 monclient(hunting): picked mon.noname-c con 0x7fe9100adc70 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:12.864 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9100b0d50 0x7fe91011edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:12.865 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 10 monclient(hunting): picked mon.noname-b con 0x7fe9100b0d50 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:12.865 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9100b4350 0x7fe9100aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:12.865 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 10 monclient(hunting): picked mon.noname-a con 0x7fe9100b4350 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:12.865 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:12.865 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9100b13e0 con 0x7fe9100b4350 2023-10-30T12:10:12.866 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:12.866 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91013ae60 con 0x7fe9100b0d50 2023-10-30T12:10:12.866 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:12.866 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9100a9dc0 con 0x7fe9100adc70 2023-10-30T12:10:12.866 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:12.867 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.844+0000 7fe917c7c640 10 monclient(hunting): authenticate will time out at 1404.058762s 2023-10-30T12:10:12.867 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe915a18640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9100adc70 0x7fe9100ad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:12.867 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe916219640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9100b4350 0x7fe9100aa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:12.867 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe915217640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9100b0d50 0x7fe91011edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:12.868 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe916219640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9100b4350 0x7fe9100aa2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42138/0 (socket says 172.21.15.3:42138) 2023-10-30T12:10:12.868 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe916219640 1 -- 172.21.15.3:0/2192156124 learned_addr learned my addr 172.21.15.3:0/2192156124 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:12.868 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe916219640 10 monclient(hunting): get_auth_request con 0x7fe9100b4350 auth_method 0 2023-10-30T12:10:12.868 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe916219640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:12.868 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe916219640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:12.869 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe916219640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:12.869 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe915a18640 10 monclient(hunting): get_auth_request con 0x7fe9100adc70 auth_method 0 2023-10-30T12:10:12.869 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe915a18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:12.869 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe915a18640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:12.869 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe915a18640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:12.870 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe915217640 10 monclient(hunting): get_auth_request con 0x7fe9100b0d50 auth_method 0 2023-10-30T12:10:12.870 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe915217640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:12.870 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe915217640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:12.870 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.845+0000 7fe915217640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:12.870 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.846+0000 7fe916219640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:12.870 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.846+0000 7fe916219640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:12.871 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.846+0000 7fe916219640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:12.871 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.846+0000 7fe915217640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:12.871 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.846+0000 7fe915217640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:12.871 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.846+0000 7fe915217640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:12.871 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.846+0000 7fe915a18640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:12.872 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.846+0000 7fe915a18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:12.872 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.846+0000 7fe915a18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:12.872 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.849+0000 7fe916219640 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2023-10-30T12:10:12.872 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.849+0000 7fe916219640 1 -- 172.21.15.3:0/2192156124 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9100adc70 msgr2=0x7fe9100ad210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:12.873 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.849+0000 7fe916219640 1 --2- 172.21.15.3:0/2192156124 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9100adc70 0x7fe9100ad210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:12.873 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.849+0000 7fe916219640 1 -- 172.21.15.3:0/2192156124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9100b0d50 msgr2=0x7fe91011edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:12.873 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.849+0000 7fe916219640 1 --2- 172.21.15.3:0/2192156124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9100b0d50 0x7fe91011edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:12.873 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.849+0000 7fe916219640 10 monclient: _finish_hunting 0 2023-10-30T12:10:12.873 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.849+0000 7fe916219640 1 monclient: found mon.noname-a 2023-10-30T12:10:12.874 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.849+0000 7fe916219640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:12.874 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.849+0000 7fe916219640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:12.874 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.849+0000 7fe916219640 1 -- 172.21.15.3:0/2192156124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91011f330 con 0x7fe9100b4350 2023-10-30T12:10:12.874 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.849+0000 7fe915217640 1 --2- 172.21.15.3:0/2192156124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9100b0d50 0x7fe91011edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:12.876 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.852+0000 7fe916219640 1 --2- 172.21.15.3:0/2192156124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9100b4350 0x7fe9100aa2e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe90800c730 tx=0x7fe90800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:12.876 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.852+0000 7fe9067fc640 10 client.?.objecter ms_handle_connect 0x7fe9100b4350 2023-10-30T12:10:12.876 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.852+0000 7fe9067fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:12.877 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.855+0000 7fe9067fc640 1 -- 172.21.15.3:0/2192156124 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe908003b70 con 0x7fe9100b4350 2023-10-30T12:10:12.877 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe9067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:12.877 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe9067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:12.877 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe9067fc640 10 monclient: dump: 2023-10-30T12:10:12.878 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:10:12.878 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:12.878 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:12.878 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:12.878 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:12.879 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:12.879 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:12.879 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:12.879 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:12.879 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:10:12.880 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe9067fc640 10 monclient: _finish_auth 0 2023-10-30T12:10:12.880 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe9067fc640 10 monclient: _check_auth_tickets 2023-10-30T12:10:12.880 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe9067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:12.880 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe9067fc640 1 -- 172.21.15.3:0/2192156124 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90800e040 con 0x7fe9100b4350 2023-10-30T12:10:12.880 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe917c7c640 5 monclient: authenticate success, global_id 4284 2023-10-30T12:10:12.881 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe9067fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:12.881 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe9067fc640 1 -- 172.21.15.3:0/2192156124 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90800ce40 con 0x7fe9100b4350 2023-10-30T12:10:12.881 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe917c7c640 10 monclient: _renew_subs 2023-10-30T12:10:12.881 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe917c7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:12.881 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe917c7c640 1 -- 172.21.15.3:0/2192156124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9100abc50 con 0x7fe9100b4350 2023-10-30T12:10:12.881 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe9067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:12.882 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe907fff640 4 set_mon_vals no callback set 2023-10-30T12:10:12.882 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe9067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:12.882 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe9067fc640 10 monclient: dump: 2023-10-30T12:10:12.882 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2023-10-30T12:10:12.882 INFO:teuthology.misc.health.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:12.883 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:12.883 INFO:teuthology.misc.health.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:12.883 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:12.883 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:12.883 INFO:teuthology.misc.health.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:12.884 INFO:teuthology.misc.health.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:12.884 INFO:teuthology.misc.health.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:12.884 INFO:teuthology.misc.health.smithi003.stderr: 2023-10-30T12:10:12.884 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe917c7c640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:12.884 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe917c7c640 10 monclient: _renew_subs 2023-10-30T12:10:12.885 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe917c7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:12.885 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe917c7c640 1 -- 172.21.15.3:0/2192156124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9100aaab0 con 0x7fe9100b4350 2023-10-30T12:10:12.885 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.856+0000 7fe917c7c640 1 librados: init done 2023-10-30T12:10:12.885 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.857+0000 7fe917c7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:12.885 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.857+0000 7fe917c7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:12.885 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.857+0000 7fe917c7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:12.886 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.857+0000 7fe917c7c640 1 -- 172.21.15.3:0/2192156124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91004e9b0 con 0x7fe9100b4350 2023-10-30T12:10:12.888 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.887+0000 7fe9067fc640 1 -- 172.21.15.3:0/2192156124 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7fe90800a850 con 0x7fe9100b4350 2023-10-30T12:10:12.889 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.888+0000 7fe9067fc640 1 --2- 172.21.15.3:0/2192156124 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe8f406c780 0x7fe8f406ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:12.890 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.888+0000 7fe915a18640 1 --2- 172.21.15.3:0/2192156124 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe8f406c780 0x7fe8f406ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:12.890 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.888+0000 7fe915a18640 10 monclient: get_auth_request con 0x7fe8f406c780 auth_method 0 2023-10-30T12:10:12.890 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.889+0000 7fe915a18640 1 --2- 172.21.15.3:0/2192156124 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe8f406c780 0x7fe8f406ec40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe900036000 tx=0x7fe90003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:12.890 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:12.889+0000 7fe9067fc640 10 client.4284.objecter ms_handle_connect 0x7fe8f406c780 2023-10-30T12:10:13.151 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.150+0000 7fe9067fc640 1 -- 172.21.15.3:0/2192156124 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3737+0+0 (secure 0 0 0) 0x7fe908058560 con 0x7fe9100b4350 2023-10-30T12:10:13.152 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.150+0000 7fe9067fc640 10 client.4284.objecter ms_dispatch 0x7fe910000b90 osd_map(18..18 src has 1..18) v4 2023-10-30T12:10:13.152 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.150+0000 7fe9067fc640 3 client.4284.objecter handle_osd_map got epochs [18,18] > 0 2023-10-30T12:10:13.152 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.150+0000 7fe9067fc640 3 client.4284.objecter handle_osd_map decoding full epoch 18 2023-10-30T12:10:13.152 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.150+0000 7fe9067fc640 20 client.4284.objecter dump_active .. 0 homeless 2023-10-30T12:10:13.153 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.150+0000 7fe9067fc640 1 -- 172.21.15.3:0/2192156124 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe90805c600 con 0x7fe9100b4350 2023-10-30T12:10:13.153 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.150+0000 7fe9067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:13.153 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.150+0000 7fe9067fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:13.335 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.333+0000 7fe917c7c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:10:13.335 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.333+0000 7fe917c7c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:10:13.335 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.333+0000 7fe917c7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:13.335 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.333+0000 7fe917c7c640 1 -- 172.21.15.3:0/2192156124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9100aaab0 con 0x7fe9100b4350 2023-10-30T12:10:13.426 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.424+0000 7fe9067fc640 1 -- 172.21.15.3:0/2192156124 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+10862 (secure 0 0 0) 0x7fe9080607b0 con 0x7fe9100b4350 2023-10-30T12:10:13.426 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.424+0000 7fe9067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:10:13.427 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.424+0000 7fe9067fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:13.427 INFO:teuthology.misc.health.smithi003.stdout: 2023-10-30T12:10:13.427 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":18,"fsid":"9bae44ae-7379-4ab2-a454-4d550607110c","created":"2023-10-30T12:09:17.680340+0000","modified":"2023-10-30T12:10:11.106790+0000","last_up_change":"2023-10-30T12:10:08.115093+0000","last_in_change":"2023-10-30T12:09:24.800567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:10:06.065827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"69005df2-28b6-4bb2-b3e6-79df0d403c70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3639283465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6809","nonce":3639283465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6811","nonce":3639283465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3639283465}]},"public_addr":"172.21.15.3:6808/3639283465","cluster_addr":"172.21.15.3:6809/3639283465","heartbeat_back_addr":"172.21.15.3:6811/3639283465","heartbeat_front_addr":"172.21.15.3:6810/3639283465","state":["exists","up"]},{"osd":1,"uuid":"6466ea5c-d74a-4709-a83a-f5d4fe17157e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":1445650280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6805","nonce":1445650280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6807","nonce":1445650280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":1445650280}]},"public_addr":"172.21.15.3:6804/1445650280","cluster_addr":"172.21.15.3:6805/1445650280","heartbeat_back_addr":"172.21.15.3:6807/1445650280","heartbeat_front_addr":"172.21.15.3:6806/1445650280","state":["exists","up"]},{"osd":2,"uuid":"49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":1407765917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6801","nonce":1407765917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6803","nonce":1407765917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":1407765917}]},"public_addr":"172.21.15.3:6800/1407765917","cluster_addr":"172.21.15.3:6801/1407765917","heartbeat_back_addr":"172.21.15.3:6803/1407765917","heartbeat_front_addr":"172.21.15.3:6802/1407765917","state":["exists","up"]},{"osd":3,"uuid":"69ec5657-9972-4f4b-8012-83886eb47429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2654726311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":2654726311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6815","nonce":2654726311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2654726311}]},"public_addr":"172.21.15.3:6812/2654726311","cluster_addr":"172.21.15.3:6813/2654726311","heartbeat_back_addr":"172.21.15.3:6815/2654726311","heartbeat_front_addr":"172.21.15.3:6814/2654726311","state":["exists","up"]},{"osd":4,"uuid":"da3dad24-4aeb-47a3-ad1d-d13c77cafad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6801","nonce":4229938171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4229938171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4229938171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6803","nonce":4229938171}]},"public_addr":"172.21.15.92:6801/4229938171","cluster_addr":"172.21.15.92:6802/4229938171","heartbeat_back_addr":"172.21.15.92:6804/4229938171","heartbeat_front_addr":"172.21.15.92:6803/4229938171","state":["exists","up"]},{"osd":5,"uuid":"d38d9d7f-b0de-48ab-a850-499c6a0a4364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6805","nonce":3208162957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3208162957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3208162957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6807","nonce":3208162957}]},"public_addr":"172.21.15.92:6805/3208162957","cluster_addr":"172.21.15.92:6806/3208162957","heartbeat_back_addr":"172.21.15.92:6808/3208162957","heartbeat_front_addr":"172.21.15.92:6807/3208162957","state":["exists","up"]},{"osd":6,"uuid":"b39d2b0a-84ef-4909-bac7-47f146305a79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":3275842310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3275842310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3275842310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6811","nonce":3275842310}]},"public_addr":"172.21.15.92:6809/3275842310","cluster_addr":"172.21.15.92:6810/3275842310","heartbeat_back_addr":"172.21.15.92:6812/3275842310","heartbeat_front_addr":"172.21.15.92:6811/3275842310","state":["exists","up"]},{"osd":7,"uuid":"0a9d81ac-5a14-4fef-877a-982594787dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":1101825516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1101825516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1101825516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":1101825516}]},"public_addr":"172.21.15.92:6813/1101825516","cluster_addr":"172.21.15.92:6814/1101825516","heartbeat_back_addr":"172.21.15.92:6816/1101825516","heartbeat_front_addr":"172.21.15.92:6815/1101825516","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.340529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.109770+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.885514+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.651007+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:02.744160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:03.156206+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:03.018599+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:03.334416+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:10:13.429 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 10 librados: watch_flush enter 2023-10-30T12:10:13.429 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 10 librados: watch_flush exit 2023-10-30T12:10:13.430 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 20 client.4284.objecter shutdown clearing up homeless session... 2023-10-30T12:10:13.430 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 10 client.4284.objecter successfully canceled tick 2023-10-30T12:10:13.430 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 1 -- 172.21.15.3:0/2192156124 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe8f406c780 msgr2=0x7fe8f406ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:13.430 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 1 --2- 172.21.15.3:0/2192156124 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe8f406c780 0x7fe8f406ec40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe900036000 tx=0x7fe90003d040 comp rx=0 tx=0).stop 2023-10-30T12:10:13.430 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 10 monclient: shutdown 2023-10-30T12:10:13.431 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:13.431 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 1 -- 172.21.15.3:0/2192156124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9100b4350 msgr2=0x7fe9100aa2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:13.431 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 1 --2- 172.21.15.3:0/2192156124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9100b4350 0x7fe9100aa2e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe90800c730 tx=0x7fe90800cbf0 comp rx=0 tx=0).stop 2023-10-30T12:10:13.431 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 1 -- 172.21.15.3:0/2192156124 shutdown_connections 2023-10-30T12:10:13.431 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 1 --2- 172.21.15.3:0/2192156124 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe8f406c780 0x7fe8f406ec40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:13.432 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 1 --2- 172.21.15.3:0/2192156124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9100b4350 0x7fe9100aa2e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:13.432 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 1 --2- 172.21.15.3:0/2192156124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9100b0d50 0x7fe91011edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:13.432 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 1 --2- 172.21.15.3:0/2192156124 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9100adc70 0x7fe9100ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:13.432 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 1 -- 172.21.15.3:0/2192156124 >> 172.21.15.3:0/2192156124 conn(0x7fe910060bf0 msgr2=0x7fe91013d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:13.432 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.427+0000 7fe917c7c640 1 -- 172.21.15.3:0/2192156124 shutdown_connections 2023-10-30T12:10:13.433 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.428+0000 7fe917c7c640 1 -- 172.21.15.3:0/2192156124 wait complete. 2023-10-30T12:10:13.433 INFO:teuthology.misc.health.smithi003.stderr:2023-10-30T12:10:13.428+0000 7fe917c7c640 1 librados: shutdown 2023-10-30T12:10:13.441 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-30T12:10:13.442 INFO:tasks.ceph:Creating RBD pool 2023-10-30T12:10:13.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-30T12:10:13.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.550+0000 7f638171d640 10 monclient: get_monmap_and_config 2023-10-30T12:10:13.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.550+0000 7f638171d640 10 monclient: build_initial_monmap 2023-10-30T12:10:13.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.550+0000 7f638171d640 10 monclient: monmap: 2023-10-30T12:10:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:13.551824+0000 2023-10-30T12:10:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:13.551824+0000 2023-10-30T12:10:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:13.554 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:13.554 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:13.554 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:13.554 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:13.554 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:13.555 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:13.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.551+0000 7f638171d640 1 Processor -- start 2023-10-30T12:10:13.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.551+0000 7f638171d640 1 -- start start 2023-10-30T12:10:13.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.551+0000 7f638171d640 10 monclient: init 2023-10-30T12:10:13.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.552+0000 7f638171d640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:13.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.552+0000 7f638171d640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-10-30T12:10:13.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.552+0000 7f638171d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f637c0b4350 0x7f637c0a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:13.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.552+0000 7f638171d640 10 monclient(hunting): picked mon.noname-b con 0x7f637c0b4350 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:13.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.552+0000 7f638171d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f637c13eb50 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:13.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.552+0000 7f638171d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f637c144f50 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:13.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.552+0000 7f638171d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637c0aa6e0 con 0x7f637c13eb50 2023-10-30T12:10:13.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.552+0000 7f638171d640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637c13f680 con 0x7f637c144f50 2023-10-30T12:10:13.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.552+0000 7f638171d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:13.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.552+0000 7f638171d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f637c0679f0 con 0x7f637c0b4350 2023-10-30T12:10:13.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.552+0000 7f638171d640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:13.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.552+0000 7f638171d640 10 monclient(hunting): authenticate will time out at 1404.766199s 2023-10-30T12:10:13.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.552+0000 7f637affd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f637c0b4350 0x7f637c0a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:13.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.552+0000 7f637b7fe640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f637c144f50 0x7f637c1453e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.3:45956/0 (socket says 172.21.15.3:45956) 2023-10-30T12:10:13.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.552+0000 7f637b7fe640 1 -- 172.21.15.3:0/1139944833 learned_addr learned my addr 172.21.15.3:0/1139944833 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:13.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.553+0000 7f637affd640 10 monclient(hunting): get_auth_request con 0x7f637c0b4350 auth_method 0 2023-10-30T12:10:13.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.553+0000 7f637affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:13.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.553+0000 7f637affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:13.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.553+0000 7f637affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:13.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.553+0000 7f637affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:13.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.553+0000 7f637affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:13.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.553+0000 7f637affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:13.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.555+0000 7f6379ffb640 1 -- 172.21.15.3:0/1139944833 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334994989 0 0) 0x7f6374001490 con 0x7f637c144f50 2023-10-30T12:10:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.555+0000 7f6379ffb640 10 monclient(hunting): _negotiate 2023-10-30T12:10:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.555+0000 7f6379ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.555+0000 7f6379ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.555+0000 7f6379ffb640 10 monclient(hunting): my global_id is 4255 2023-10-30T12:10:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.555+0000 7f6379ffb640 1 -- 172.21.15.3:0/1139944833 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6364003520 con 0x7f637c144f50 2023-10-30T12:10:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.555+0000 7f637affd640 10 monclient(hunting): handle_auth_done global_id 4208 payload 274 2023-10-30T12:10:13.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.555+0000 7f637affd640 1 -- 172.21.15.3:0/1139944833 >> v1:172.21.15.92:6789/0 conn(0x7f637c144f50 legacy=0x7f637c1453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:13.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.555+0000 7f637affd640 1 -- 172.21.15.3:0/1139944833 >> v1:172.21.15.3:6789/0 conn(0x7f637c13eb50 legacy=0x7f637c13ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:13.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.555+0000 7f637affd640 10 monclient: _finish_hunting 0 2023-10-30T12:10:13.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.555+0000 7f637affd640 1 monclient: found mon.noname-b 2023-10-30T12:10:13.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.555+0000 7f637affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:13.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.555+0000 7f637affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:10:13.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.555+0000 7f637affd640 1 -- 172.21.15.3:0/1139944833 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c0a8910 con 0x7f637c0b4350 2023-10-30T12:10:13.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.557+0000 7f637affd640 1 --2- 172.21.15.3:0/1139944833 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f637c0b4350 0x7f637c0a83d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f63700097d0 tx=0x7f637002b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc1a05ff309b4f16 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:13.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f6379ffb640 1 -- 172.21.15.3:0/1139944833 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6370031770 con 0x7f637c0b4350 2023-10-30T12:10:13.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f6379ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:13.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f6379ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:13.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f6379ffb640 10 monclient: dump: 2023-10-30T12:10:13.571 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:13.571 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:13.571 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:13.571 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:13.572 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:13.572 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:13.572 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:13.572 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:13.572 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:13.572 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:13.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f6379ffb640 10 monclient: _finish_auth 0 2023-10-30T12:10:13.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f6379ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:10:13.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f6379ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:13.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f6379ffb640 1 -- 172.21.15.3:0/1139944833 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63700318e0 con 0x7f637c0b4350 2023-10-30T12:10:13.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f6379ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:13.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f6379ffb640 1 -- 172.21.15.3:0/1139944833 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6370031c80 con 0x7f637c0b4350 2023-10-30T12:10:13.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f638171d640 5 monclient: authenticate success, global_id 4208 2023-10-30T12:10:13.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f638171d640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:13.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f638171d640 4 set_mon_vals no callback set 2023-10-30T12:10:13.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f6379ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:13.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f6379ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:13.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.561+0000 7f6379ffb640 10 monclient: dump: 2023-10-30T12:10:13.575 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:13.575 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:13.575 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:13.575 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:13.576 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:13.576 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:13.576 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:13.576 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:13.576 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:13.577 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:13.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.562+0000 7f638171d640 10 monclient: shutdown 2023-10-30T12:10:13.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.562+0000 7f638171d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:13.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.562+0000 7f638171d640 1 -- 172.21.15.3:0/1139944833 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f637c0b4350 msgr2=0x7f637c0a83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:13.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.562+0000 7f638171d640 1 --2- 172.21.15.3:0/1139944833 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f637c0b4350 0x7f637c0a83d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f63700097d0 tx=0x7f637002b810 comp rx=0 tx=0).stop 2023-10-30T12:10:13.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.562+0000 7f638171d640 1 -- 172.21.15.3:0/1139944833 shutdown_connections 2023-10-30T12:10:13.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.562+0000 7f638171d640 1 --2- 172.21.15.3:0/1139944833 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f637c0b4350 0x7f637c0a83d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:13.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.562+0000 7f638171d640 1 -- 172.21.15.3:0/1139944833 >> 172.21.15.3:0/1139944833 conn(0x7f637c065ea0 msgr2=0x7f637c0662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:13.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.562+0000 7f638171d640 1 -- 172.21.15.3:0/1139944833 shutdown_connections 2023-10-30T12:10:13.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.562+0000 7f638171d640 1 -- 172.21.15.3:0/1139944833 wait complete. 2023-10-30T12:10:13.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.563+0000 7f638171d640 10 monclient: build_initial_monmap 2023-10-30T12:10:13.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.563+0000 7f638171d640 10 monclient: monmap: 2023-10-30T12:10:13.579 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:13.579 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:13.579 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:13.579 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:13.580 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:13.580 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:13.580 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:13.580 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:13.580 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:13.581 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:13.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.563+0000 7f638171d640 1 librados: starting msgr at 2023-10-30T12:10:13.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.563+0000 7f638171d640 1 librados: starting objecter 2023-10-30T12:10:13.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 1 Processor -- start 2023-10-30T12:10:13.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 1 -- start start 2023-10-30T12:10:13.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 1 librados: setting wanted keys 2023-10-30T12:10:13.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 1 librados: calling monclient init 2023-10-30T12:10:13.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 10 monclient: init 2023-10-30T12:10:13.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:13.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:13.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f637c0b4350 0x7f637c0b7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:13.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 10 monclient(hunting): picked mon.noname-b con 0x7f637c0b4350 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:13.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f637c13eb50 0x7f637c13f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:13.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 10 monclient(hunting): picked mon.noname-a con 0x7f637c13eb50 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:13.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f637c144f50 0x7f637c141b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:13.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 10 monclient(hunting): picked mon.noname-c con 0x7f637c144f50 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:13.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:13.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c145470 con 0x7f637c13eb50 2023-10-30T12:10:13.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:13.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c141210 con 0x7f637c0b4350 2023-10-30T12:10:13.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:13.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f637c141480 con 0x7f637c144f50 2023-10-30T12:10:13.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:13.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.564+0000 7f638171d640 10 monclient(hunting): authenticate will time out at 1404.778857s 2023-10-30T12:10:13.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.565+0000 7f637affd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f637c0b4350 0x7f637c0b7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:13.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.565+0000 7f637b7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f637c144f50 0x7f637c141b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:13.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.565+0000 7f637affd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f637c0b4350 0x7f637c0b7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56176/0 (socket says 172.21.15.3:56176) 2023-10-30T12:10:13.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.565+0000 7f637affd640 1 -- 172.21.15.3:0/2721882104 learned_addr learned my addr 172.21.15.3:0/2721882104 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:13.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.565+0000 7f637affd640 10 monclient(hunting): get_auth_request con 0x7f637c0b4350 auth_method 0 2023-10-30T12:10:13.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.565+0000 7f637affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:13.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.565+0000 7f637affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:13.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.565+0000 7f637affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:13.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.566+0000 7f637b7fe640 10 monclient(hunting): get_auth_request con 0x7f637c144f50 auth_method 0 2023-10-30T12:10:13.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.566+0000 7f637b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:13.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.566+0000 7f637b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:13.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.566+0000 7f637b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:13.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.565+0000 7f637a7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f637c13eb50 0x7f637c13f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:13.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.566+0000 7f637a7fc640 10 monclient(hunting): get_auth_request con 0x7f637c13eb50 auth_method 0 2023-10-30T12:10:13.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.566+0000 7f637a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:13.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.566+0000 7f637a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:13.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.566+0000 7f637a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:13.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.566+0000 7f637affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:13.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.566+0000 7f637affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:13.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.566+0000 7f637affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:13.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.566+0000 7f637b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:13.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.566+0000 7f637b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:13.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.567+0000 7f637b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:13.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.567+0000 7f637a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:13.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.567+0000 7f637a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:13.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.567+0000 7f637a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:13.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.569+0000 7f637affd640 10 monclient(hunting): handle_auth_done global_id 4258 payload 824 2023-10-30T12:10:13.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.569+0000 7f637affd640 1 -- 172.21.15.3:0/2721882104 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f637c144f50 msgr2=0x7f637c141b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:13.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.569+0000 7f637affd640 1 --2- 172.21.15.3:0/2721882104 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f637c144f50 0x7f637c141b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:13.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.569+0000 7f637affd640 1 -- 172.21.15.3:0/2721882104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f637c13eb50 msgr2=0x7f637c13f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:13.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.569+0000 7f637affd640 1 --2- 172.21.15.3:0/2721882104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f637c13eb50 0x7f637c13f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:13.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.569+0000 7f637affd640 10 monclient: _finish_hunting 0 2023-10-30T12:10:13.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.569+0000 7f637affd640 1 monclient: found mon.noname-b 2023-10-30T12:10:13.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.569+0000 7f637affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:13.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.569+0000 7f637affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:13.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.569+0000 7f637affd640 1 -- 172.21.15.3:0/2721882104 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c13fe70 con 0x7f637c0b4350 2023-10-30T12:10:13.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.572+0000 7f637affd640 1 --2- 172.21.15.3:0/2721882104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f637c0b4350 0x7f637c0b7b50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6370034040 tx=0x7f63700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:13.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.572+0000 7f63637fe640 10 client.?.objecter ms_handle_connect 0x7f637c0b4350 2023-10-30T12:10:13.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.572+0000 7f63637fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:13.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f63637fe640 1 -- 172.21.15.3:0/2721882104 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6370032810 con 0x7f637c0b4350 2023-10-30T12:10:13.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f63637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:13.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f63637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:13.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f63637fe640 10 monclient: dump: 2023-10-30T12:10:13.595 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:13.596 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:13.596 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:13.596 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:13.596 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:13.596 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:13.597 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:13.597 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:13.597 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:13.597 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:13.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f63637fe640 10 monclient: _finish_auth 0 2023-10-30T12:10:13.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f63637fe640 10 monclient: _check_auth_tickets 2023-10-30T12:10:13.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f63637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:13.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f63637fe640 1 -- 172.21.15.3:0/2721882104 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637003f030 con 0x7f637c0b4350 2023-10-30T12:10:13.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f63637fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:13.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f638171d640 5 monclient: authenticate success, global_id 4258 2023-10-30T12:10:13.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f63637fe640 1 -- 172.21.15.3:0/2721882104 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6370032c00 con 0x7f637c0b4350 2023-10-30T12:10:13.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f63637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:13.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f63637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:13.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f63637fe640 10 monclient: dump: 2023-10-30T12:10:13.599 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:13.599 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:13.600 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:13.600 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:13.600 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:13.600 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:13.600 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:13.601 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:13.601 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:13.601 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:13.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f638171d640 10 monclient: _renew_subs 2023-10-30T12:10:13.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f638171d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:13.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f638171d640 1 -- 172.21.15.3:0/2721882104 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f637c140390 con 0x7f637c0b4350 2023-10-30T12:10:13.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f63797fa640 4 set_mon_vals no callback set 2023-10-30T12:10:13.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f638171d640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:13.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f638171d640 10 monclient: _renew_subs 2023-10-30T12:10:13.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f638171d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:13.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f638171d640 1 -- 172.21.15.3:0/2721882104 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f637c142070 con 0x7f637c0b4350 2023-10-30T12:10:13.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.576+0000 7f638171d640 1 librados: init done 2023-10-30T12:10:13.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.577+0000 7f638171d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:13.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.577+0000 7f638171d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:13.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.577+0000 7f638171d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:13.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.577+0000 7f638171d640 1 -- 172.21.15.3:0/2721882104 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f637c0a8bc0 con 0x7f637c0b4350 2023-10-30T12:10:13.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.815+0000 7f63637fe640 1 -- 172.21.15.3:0/2721882104 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 89138+0+0 (secure 0 0 0) 0x7f637004e450 con 0x7f637c0b4350 2023-10-30T12:10:13.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.816+0000 7f63637fe640 1 --2- 172.21.15.3:0/2721882104 >> v2:172.21.15.92:6800/1750155735 conn(0x7f636806c780 0x7f636806ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:13.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.816+0000 7f637a7fc640 1 --2- 172.21.15.3:0/2721882104 >> v2:172.21.15.92:6800/1750155735 conn(0x7f636806c780 0x7f636806ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:13.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.816+0000 7f637a7fc640 10 monclient: get_auth_request con 0x7f636806c780 auth_method 0 2023-10-30T12:10:13.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.816+0000 7f637a7fc640 1 --2- 172.21.15.3:0/2721882104 >> v2:172.21.15.92:6800/1750155735 conn(0x7f636806c780 0x7f636806ec40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f637c13d860 tx=0x7f636c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:13.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.816+0000 7f63637fe640 10 client.4258.objecter ms_handle_connect 0x7f636806c780 2023-10-30T12:10:13.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.819+0000 7f63637fe640 1 -- 172.21.15.3:0/2721882104 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3737+0+0 (secure 0 0 0) 0x7f637002f050 con 0x7f637c0b4350 2023-10-30T12:10:13.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.820+0000 7f63637fe640 10 client.4258.objecter ms_dispatch 0x7f637c000b90 osd_map(18..18 src has 1..18) v4 2023-10-30T12:10:13.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.820+0000 7f63637fe640 3 client.4258.objecter handle_osd_map got epochs [18,18] > 0 2023-10-30T12:10:13.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.820+0000 7f63637fe640 3 client.4258.objecter handle_osd_map decoding full epoch 18 2023-10-30T12:10:13.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.820+0000 7f63637fe640 20 client.4258.objecter dump_active .. 0 homeless 2023-10-30T12:10:13.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.829+0000 7f63637fe640 1 -- 172.21.15.3:0/2721882104 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6370037070 con 0x7f637c0b4350 2023-10-30T12:10:13.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.829+0000 7f63637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:13.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:13.829+0000 7f63637fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:14.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:14.036+0000 7f638171d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-10-30T12:10:14.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:14.036+0000 7f638171d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-10-30T12:10:14.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:14.036+0000 7f638171d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:14.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:14.036+0000 7f638171d640 1 -- 172.21.15.3:0/2721882104 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f637c0a8910 con 0x7f637c0b4350 2023-10-30T12:10:14.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:14.324+0000 7f63637fe640 1 -- 172.21.15.3:0/2721882104 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f6370075b10 con 0x7f637c0b4350 2023-10-30T12:10:15.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.241+0000 7f63637fe640 1 -- 172.21.15.3:0/2721882104 <== mon.1 v2:172.21.15.92:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v19) v1 ==== 109+0+0 (secure 0 0 0) 0x7f63700093c0 con 0x7f637c0b4350 2023-10-30T12:10:15.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.241+0000 7f63637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-10-30T12:10:15.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.241+0000 7f63637fe640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-10-30T12:10:15.243 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'rbd' created 2023-10-30T12:10:15.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 10 librados: watch_flush enter 2023-10-30T12:10:15.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 10 librados: watch_flush exit 2023-10-30T12:10:15.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 20 client.4258.objecter shutdown clearing up homeless session... 2023-10-30T12:10:15.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 10 client.4258.objecter successfully canceled tick 2023-10-30T12:10:15.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 1 -- 172.21.15.3:0/2721882104 >> v2:172.21.15.92:6800/1750155735 conn(0x7f636806c780 msgr2=0x7f636806ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:15.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 1 --2- 172.21.15.3:0/2721882104 >> v2:172.21.15.92:6800/1750155735 conn(0x7f636806c780 0x7f636806ec40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f637c13d860 tx=0x7f636c009040 comp rx=0 tx=0).stop 2023-10-30T12:10:15.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 10 monclient: shutdown 2023-10-30T12:10:15.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:15.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 1 -- 172.21.15.3:0/2721882104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f637c0b4350 msgr2=0x7f637c0b7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:15.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 1 --2- 172.21.15.3:0/2721882104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f637c0b4350 0x7f637c0b7b50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6370034040 tx=0x7f63700046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:15.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 1 -- 172.21.15.3:0/2721882104 shutdown_connections 2023-10-30T12:10:15.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 1 --2- 172.21.15.3:0/2721882104 >> v2:172.21.15.92:6800/1750155735 conn(0x7f636806c780 0x7f636806ec40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:15.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 1 --2- 172.21.15.3:0/2721882104 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f637c144f50 0x7f637c141b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:15.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 1 --2- 172.21.15.3:0/2721882104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f637c13eb50 0x7f637c13f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:15.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 1 --2- 172.21.15.3:0/2721882104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f637c0b4350 0x7f637c0b7b50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:15.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 1 -- 172.21.15.3:0/2721882104 >> 172.21.15.3:0/2721882104 conn(0x7f637c065ea0 msgr2=0x7f637c144840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:15.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 1 -- 172.21.15.3:0/2721882104 shutdown_connections 2023-10-30T12:10:15.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 1 -- 172.21.15.3:0/2721882104 wait complete. 2023-10-30T12:10:15.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.243+0000 7f638171d640 1 librados: shutdown 2023-10-30T12:10:15.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-30T12:10:15.369 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:15.304 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:15.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.393+0000 7fdc7f7c5640 10 monclient: get_monmap_and_config 2023-10-30T12:10:15.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.393+0000 7fdc7f7c5640 10 monclient: build_initial_monmap 2023-10-30T12:10:15.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.393+0000 7fdc7f7c5640 10 monclient: monmap: 2023-10-30T12:10:15.395 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:15.395 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:15.396 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:15.394361+0000 2023-10-30T12:10:15.396 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:15.394361+0000 2023-10-30T12:10:15.396 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:15.396 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:15.396 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:15.397 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:15.397 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:15.397 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:15.397 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:15.397 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:15.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.393+0000 7fdc7f7c5640 1 Processor -- start 2023-10-30T12:10:15.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.394+0000 7fdc7f7c5640 1 -- start start 2023-10-30T12:10:15.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.394+0000 7fdc7f7c5640 10 monclient: init 2023-10-30T12:10:15.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.394+0000 7fdc7f7c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:15.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.394+0000 7fdc7f7c5640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2023-10-30T12:10:15.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.394+0000 7fdc7f7c5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdc780b4360 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:15.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.394+0000 7fdc7f7c5640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdc7813f320 0x7fdc78144e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:15.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.394+0000 7fdc7f7c5640 10 monclient(hunting): picked mon.noname-a con 0x7fdc7813f320 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:15.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.394+0000 7fdc7f7c5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdc780b12b0 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:15.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.394+0000 7fdc7f7c5640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc780aa180 con 0x7fdc780b4360 2023-10-30T12:10:15.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.394+0000 7fdc7f7c5640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc780aa3d0 con 0x7fdc780b12b0 2023-10-30T12:10:15.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.394+0000 7fdc7f7c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:15.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.394+0000 7fdc7f7c5640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc78067a30 con 0x7fdc7813f320 2023-10-30T12:10:15.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.394+0000 7fdc7f7c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:15.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.394+0000 7fdc7f7c5640 10 monclient(hunting): authenticate will time out at 1406.608910s 2023-10-30T12:10:15.416 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:15.351 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:15.433 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:15.368 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:15.465 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:15.373 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:15.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.468+0000 7fdc7dd62640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fdc780b12b0 0x7fdc78145350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.3:45958/0 (socket says 172.21.15.3:45958) 2023-10-30T12:10:15.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.468+0000 7fdc7dd62640 1 -- 172.21.15.3:0/2581342711 learned_addr learned my addr 172.21.15.3:0/2581342711 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:15.488 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:15.424 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:15.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.487+0000 7fdc7cd60640 1 --2- 172.21.15.3:0/2581342711 >> v2:172.21.15.3:3300/0 conn(0x7fdc7813f320 0x7fdc78144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:15.534 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:15.442 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:15.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.546+0000 7fdc7cd60640 10 monclient(hunting): get_auth_request con 0x7fdc7813f320 auth_method 0 2023-10-30T12:10:15.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.546+0000 7fdc7cd60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:15.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.546+0000 7fdc7cd60640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:15.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.546+0000 7fdc7cd60640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:15.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.564+0000 7fdc7cd60640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:15.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.564+0000 7fdc7cd60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:15.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.564+0000 7fdc7cd60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:15.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.566+0000 7fdc7cd60640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2023-10-30T12:10:15.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.566+0000 7fdc7cd60640 1 -- 172.21.15.3:0/2581342711 >> v1:172.21.15.92:6789/0 conn(0x7fdc780b12b0 legacy=0x7fdc78145350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:15.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.566+0000 7fdc7cd60640 1 -- 172.21.15.3:0/2581342711 >> v1:172.21.15.3:6789/0 conn(0x7fdc780b4360 legacy=0x7fdc7813ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.566+0000 7fdc7cd60640 10 monclient: _finish_hunting 0 2023-10-30T12:10:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.566+0000 7fdc7cd60640 1 monclient: found mon.noname-a 2023-10-30T12:10:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.566+0000 7fdc7cd60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.566+0000 7fdc7cd60640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.566+0000 7fdc7cd60640 1 -- 172.21.15.3:0/2581342711 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc780aa6b0 con 0x7fdc7813f320 2023-10-30T12:10:15.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.634+0000 7fdc7cd60640 1 --2- 172.21.15.3:0/2581342711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc7813f320 0x7fdc78144e10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdc680097a0 tx=0x7fdc6802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77cca559d9c69d25 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:15.727 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:15.663 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:15.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc6ffff640 1 -- 172.21.15.3:0/2581342711 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc680315a0 con 0x7fdc7813f320 2023-10-30T12:10:15.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:15.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc6ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:15.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc6ffff640 10 monclient: dump: 2023-10-30T12:10:15.777 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:15.777 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:15.777 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:15.777 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:15.777 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:15.778 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:15.778 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:15.778 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:15.778 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:15.778 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:15.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc6ffff640 10 monclient: _finish_auth 0 2023-10-30T12:10:15.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc6ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:15.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:15.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc6ffff640 1 -- 172.21.15.3:0/2581342711 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc68031710 con 0x7fdc7813f320 2023-10-30T12:10:15.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc6ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:15.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc6ffff640 1 -- 172.21.15.3:0/2581342711 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68031ab0 con 0x7fdc7813f320 2023-10-30T12:10:15.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:15.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:15.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc6ffff640 10 monclient: dump: 2023-10-30T12:10:15.780 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:15.780 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:15.781 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:15.781 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:15.781 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:15.781 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:15.781 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:15.782 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:15.782 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:15.782 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:15.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc7f7c5640 5 monclient: authenticate success, global_id 4293 2023-10-30T12:10:15.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc7f7c5640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:15.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.774+0000 7fdc7f7c5640 4 set_mon_vals no callback set 2023-10-30T12:10:15.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.775+0000 7fdc7f7c5640 10 monclient: shutdown 2023-10-30T12:10:15.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.775+0000 7fdc7f7c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:15.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.775+0000 7fdc7f7c5640 1 -- 172.21.15.3:0/2581342711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc7813f320 msgr2=0x7fdc78144e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:15.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.775+0000 7fdc7f7c5640 1 --2- 172.21.15.3:0/2581342711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc7813f320 0x7fdc78144e10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdc680097a0 tx=0x7fdc6802b840 comp rx=0 tx=0).stop 2023-10-30T12:10:15.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.775+0000 7fdc7f7c5640 1 -- 172.21.15.3:0/2581342711 shutdown_connections 2023-10-30T12:10:15.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.775+0000 7fdc7f7c5640 1 --2- 172.21.15.3:0/2581342711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc7813f320 0x7fdc78144e10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:15.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.775+0000 7fdc7f7c5640 1 -- 172.21.15.3:0/2581342711 >> 172.21.15.3:0/2581342711 conn(0x7fdc78065ed0 msgr2=0x7fdc780662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:15.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.775+0000 7fdc7f7c5640 1 -- 172.21.15.3:0/2581342711 shutdown_connections 2023-10-30T12:10:15.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.776+0000 7fdc7f7c5640 1 -- 172.21.15.3:0/2581342711 wait complete. 2023-10-30T12:10:15.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.776+0000 7fdc7f7c5640 10 monclient: build_initial_monmap 2023-10-30T12:10:15.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.776+0000 7fdc7f7c5640 10 monclient: monmap: 2023-10-30T12:10:15.787 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:15.787 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:15.787 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:15.788 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:15.788 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:15.788 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:15.788 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:15.788 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:15.789 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:15.789 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:15.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.777+0000 7fdc7f7c5640 1 librados: starting msgr at 2023-10-30T12:10:15.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.777+0000 7fdc7f7c5640 1 librados: starting objecter 2023-10-30T12:10:15.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.777+0000 7fdc7f7c5640 1 Processor -- start 2023-10-30T12:10:15.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.777+0000 7fdc7f7c5640 1 -- start start 2023-10-30T12:10:15.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.777+0000 7fdc7f7c5640 1 librados: setting wanted keys 2023-10-30T12:10:15.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.777+0000 7fdc7f7c5640 1 librados: calling monclient init 2023-10-30T12:10:15.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.777+0000 7fdc7f7c5640 10 monclient: init 2023-10-30T12:10:15.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:15.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:10:15.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc780b12b0 0x7fdc7813dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:15.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 10 monclient(hunting): picked mon.noname-a con 0x7fdc780b12b0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:15.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc780b4360 0x7fdc78142d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:15.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 10 monclient(hunting): picked mon.noname-b con 0x7fdc780b4360 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:15.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc7813d110 0x7fdc78143250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:15.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 10 monclient(hunting): picked mon.noname-c con 0x7fdc7813d110 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:15.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:15.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc780b1020 con 0x7fdc780b12b0 2023-10-30T12:10:15.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:15.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7813e240 con 0x7fdc780b4360 2023-10-30T12:10:15.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:15.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7813e500 con 0x7fdc7813d110 2023-10-30T12:10:15.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:15.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.778+0000 7fdc7f7c5640 10 monclient(hunting): authenticate will time out at 1406.992901s 2023-10-30T12:10:15.794 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:15.720 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:15.810 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:15.718 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:15.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.817+0000 7fdc7d561640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc780b12b0 0x7fdc7813dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:15.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.817+0000 7fdc7dd62640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc7813d110 0x7fdc78143250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:15.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.818+0000 7fdc7d561640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc780b12b0 0x7fdc7813dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42158/0 (socket says 172.21.15.3:42158) 2023-10-30T12:10:15.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.818+0000 7fdc7d561640 1 -- 172.21.15.3:0/708121781 learned_addr learned my addr 172.21.15.3:0/708121781 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:15.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.818+0000 7fdc7d561640 10 monclient(hunting): get_auth_request con 0x7fdc780b12b0 auth_method 0 2023-10-30T12:10:15.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.818+0000 7fdc7d561640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:15.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.818+0000 7fdc7d561640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:15.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.818+0000 7fdc7d561640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:15.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.820+0000 7fdc7d561640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:15.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.820+0000 7fdc7d561640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:15.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.820+0000 7fdc7d561640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:15.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.823+0000 7fdc7cd60640 1 --2- 172.21.15.3:0/708121781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc780b4360 0x7fdc78142d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:15.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.824+0000 7fdc7d561640 10 monclient(hunting): handle_auth_done global_id 4296 payload 824 2023-10-30T12:10:15.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.824+0000 7fdc7d561640 1 -- 172.21.15.3:0/708121781 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc7813d110 msgr2=0x7fdc78143250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:15.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.824+0000 7fdc7d561640 1 --2- 172.21.15.3:0/708121781 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc7813d110 0x7fdc78143250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:15.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.824+0000 7fdc7d561640 1 -- 172.21.15.3:0/708121781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc780b4360 msgr2=0x7fdc78142d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:15.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.824+0000 7fdc7d561640 1 --2- 172.21.15.3:0/708121781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc780b4360 0x7fdc78142d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:15.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.824+0000 7fdc7d561640 10 monclient: _finish_hunting 0 2023-10-30T12:10:15.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.824+0000 7fdc7d561640 1 monclient: found mon.noname-a 2023-10-30T12:10:15.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.824+0000 7fdc7d561640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:15.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.824+0000 7fdc7d561640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:15.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.824+0000 7fdc7d561640 1 -- 172.21.15.3:0/708121781 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc78144a90 con 0x7fdc780b12b0 2023-10-30T12:10:15.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.842+0000 7fdc7d561640 1 --2- 172.21.15.3:0/708121781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc780b12b0 0x7fdc7813dd00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdc7400c460 tx=0x7fdc7400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:15.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.842+0000 7fdc6dffb640 10 client.?.objecter ms_handle_connect 0x7fdc780b12b0 2023-10-30T12:10:15.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.842+0000 7fdc6dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:15.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.865+0000 7fdc6dffb640 1 -- 172.21.15.3:0/708121781 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc74011790 con 0x7fdc780b12b0 2023-10-30T12:10:15.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.865+0000 7fdc6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:15.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.865+0000 7fdc6dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:15.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.865+0000 7fdc6dffb640 10 monclient: dump: 2023-10-30T12:10:15.869 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:15.869 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:15.869 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:15.869 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:15.869 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:15.870 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:15.870 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:15.870 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:15.870 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:15.871 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:15.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc6dffb640 10 monclient: _finish_auth 0 2023-10-30T12:10:15.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc6dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:10:15.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:15.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc6dffb640 1 -- 172.21.15.3:0/708121781 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7400e040 con 0x7fdc780b12b0 2023-10-30T12:10:15.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc7f7c5640 5 monclient: authenticate success, global_id 4296 2023-10-30T12:10:15.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc7f7c5640 10 monclient: _renew_subs 2023-10-30T12:10:15.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc7f7c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:15.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc7f7c5640 1 -- 172.21.15.3:0/708121781 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7813c700 con 0x7fdc780b12b0 2023-10-30T12:10:15.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc6dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:15.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc6dffb640 1 -- 172.21.15.3:0/708121781 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc74011b80 con 0x7fdc780b12b0 2023-10-30T12:10:15.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:15.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc6dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:15.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc6dffb640 10 monclient: dump: 2023-10-30T12:10:15.873 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:15.873 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:15.873 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:15.874 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:15.874 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:15.874 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:15.874 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:15.874 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:15.875 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:15.875 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:15.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc6f7fe640 4 set_mon_vals no callback set 2023-10-30T12:10:15.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc7f7c5640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:15.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc7f7c5640 10 monclient: _renew_subs 2023-10-30T12:10:15.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc7f7c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:15.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc7f7c5640 1 -- 172.21.15.3:0/708121781 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc781439c0 con 0x7fdc780b12b0 2023-10-30T12:10:15.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.866+0000 7fdc7f7c5640 1 librados: init done 2023-10-30T12:10:15.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.867+0000 7fdc7f7c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:15.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.867+0000 7fdc7f7c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:15.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.867+0000 7fdc7f7c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:15.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:15.867+0000 7fdc7f7c5640 1 -- 172.21.15.3:0/708121781 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc780753b0 con 0x7fdc780b12b0 2023-10-30T12:10:16.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.159+0000 7fdc6dffb640 1 -- 172.21.15.3:0/708121781 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fdc74011cf0 con 0x7fdc780b12b0 2023-10-30T12:10:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.160+0000 7fdc6dffb640 1 --2- 172.21.15.3:0/708121781 >> v2:172.21.15.92:6800/1750155735 conn(0x7fdc5c06c810 0x7fdc5c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.160+0000 7fdc7cd60640 1 --2- 172.21.15.3:0/708121781 >> v2:172.21.15.92:6800/1750155735 conn(0x7fdc5c06c810 0x7fdc5c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.160+0000 7fdc7cd60640 10 monclient: get_auth_request con 0x7fdc5c06c810 auth_method 0 2023-10-30T12:10:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.161+0000 7fdc7cd60640 1 --2- 172.21.15.3:0/708121781 >> v2:172.21.15.92:6800/1750155735 conn(0x7fdc5c06c810 0x7fdc5c06ecd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdc68036000 tx=0x7fdc6803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:16.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.161+0000 7fdc6dffb640 10 client.4296.objecter ms_handle_connect 0x7fdc5c06c810 2023-10-30T12:10:16.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.175+0000 7fdc6dffb640 1 -- 172.21.15.3:0/708121781 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4079+0+0 (secure 0 0 0) 0x7fdc74058dd0 con 0x7fdc780b12b0 2023-10-30T12:10:16.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.175+0000 7fdc6dffb640 10 client.4296.objecter ms_dispatch 0x7fdc78000b90 osd_map(19..19 src has 1..19) v4 2023-10-30T12:10:16.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.175+0000 7fdc6dffb640 3 client.4296.objecter handle_osd_map got epochs [19,19] > 0 2023-10-30T12:10:16.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.175+0000 7fdc6dffb640 3 client.4296.objecter handle_osd_map decoding full epoch 19 2023-10-30T12:10:16.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.175+0000 7fdc6dffb640 20 client.4296.objecter dump_active .. 0 homeless 2023-10-30T12:10:16.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.175+0000 7fdc6dffb640 1 -- 172.21.15.3:0/708121781 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdc74017070 con 0x7fdc780b12b0 2023-10-30T12:10:16.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.175+0000 7fdc6dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:16.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.175+0000 7fdc6dffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:16.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.362+0000 7fdc7f7c5640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-10-30T12:10:16.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.362+0000 7fdc7f7c5640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-10-30T12:10:16.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.362+0000 7fdc7f7c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:16.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:16.362+0000 7fdc7f7c5640 1 -- 172.21.15.3:0/708121781 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdc781439c0 con 0x7fdc780b12b0 2023-10-30T12:10:18.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.010+0000 7fdc6dffb640 1 -- 172.21.15.3:0/708121781 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v21) v1 ==== 173+0+0 (secure 0 0 0) 0x7fdc740586d0 con 0x7fdc780b12b0 2023-10-30T12:10:18.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.010+0000 7fdc6dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-10-30T12:10:18.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.010+0000 7fdc6dffb640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-10-30T12:10:18.013 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-30T12:10:18.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.012+0000 7fdc7f7c5640 10 librados: watch_flush enter 2023-10-30T12:10:18.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.012+0000 7fdc7f7c5640 10 librados: watch_flush exit 2023-10-30T12:10:18.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.012+0000 7fdc7f7c5640 20 client.4296.objecter shutdown clearing up homeless session... 2023-10-30T12:10:18.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.012+0000 7fdc7f7c5640 10 client.4296.objecter successfully canceled tick 2023-10-30T12:10:18.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.012+0000 7fdc7f7c5640 1 -- 172.21.15.3:0/708121781 >> v2:172.21.15.92:6800/1750155735 conn(0x7fdc5c06c810 msgr2=0x7fdc5c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:18.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.012+0000 7fdc7f7c5640 1 --2- 172.21.15.3:0/708121781 >> v2:172.21.15.92:6800/1750155735 conn(0x7fdc5c06c810 0x7fdc5c06ecd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdc68036000 tx=0x7fdc6803d040 comp rx=0 tx=0).stop 2023-10-30T12:10:18.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.012+0000 7fdc7f7c5640 10 monclient: shutdown 2023-10-30T12:10:18.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.012+0000 7fdc7f7c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:18.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.012+0000 7fdc7f7c5640 1 -- 172.21.15.3:0/708121781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc780b12b0 msgr2=0x7fdc7813dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:18.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.012+0000 7fdc7f7c5640 1 --2- 172.21.15.3:0/708121781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc780b12b0 0x7fdc7813dd00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdc7400c460 tx=0x7fdc7400c920 comp rx=0 tx=0).stop 2023-10-30T12:10:18.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.013+0000 7fdc7f7c5640 1 -- 172.21.15.3:0/708121781 shutdown_connections 2023-10-30T12:10:18.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.013+0000 7fdc7f7c5640 1 --2- 172.21.15.3:0/708121781 >> v2:172.21.15.92:6800/1750155735 conn(0x7fdc5c06c810 0x7fdc5c06ecd0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.013+0000 7fdc7f7c5640 1 --2- 172.21.15.3:0/708121781 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc7813d110 0x7fdc78143250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.013+0000 7fdc7f7c5640 1 --2- 172.21.15.3:0/708121781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc780b4360 0x7fdc78142d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.013+0000 7fdc7f7c5640 1 --2- 172.21.15.3:0/708121781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc780b12b0 0x7fdc7813dd00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.013+0000 7fdc7f7c5640 1 -- 172.21.15.3:0/708121781 >> 172.21.15.3:0/708121781 conn(0x7fdc78065ed0 msgr2=0x7fdc780662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:18.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.013+0000 7fdc7f7c5640 1 -- 172.21.15.3:0/708121781 shutdown_connections 2023-10-30T12:10:18.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.013+0000 7fdc7f7c5640 1 -- 172.21.15.3:0/708121781 wait complete. 2023-10-30T12:10:18.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.013+0000 7fdc7f7c5640 1 librados: shutdown 2023-10-30T12:10:18.026 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-30T12:10:18.027 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-30T12:10:18.027 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-30T12:10:18.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-30T12:10:18.028 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-30T12:10:18.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.140+0000 7f95eb77c640 10 monclient: get_monmap_and_config 2023-10-30T12:10:18.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.140+0000 7f95eb77c640 10 monclient: build_initial_monmap 2023-10-30T12:10:18.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.140+0000 7f95eb77c640 10 monclient: monmap: 2023-10-30T12:10:18.143 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:18.143 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:18.143 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:18.141644+0000 2023-10-30T12:10:18.143 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:18.141644+0000 2023-10-30T12:10:18.144 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:18.144 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:18.144 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:18.144 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:18.144 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:18.145 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:18.145 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:18.145 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:18.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.141+0000 7f95eb77c640 1 Processor -- start 2023-10-30T12:10:18.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.141+0000 7f95eb77c640 1 -- start start 2023-10-30T12:10:18.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.141+0000 7f95eb77c640 10 monclient: init 2023-10-30T12:10:18.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.142+0000 7f95eb77c640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:18.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.142+0000 7f95eb77c640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-10-30T12:10:18.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.142+0000 7f95eb77c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f95e40b4390 0x7f95e40a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:18.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.142+0000 7f95eb77c640 10 monclient(hunting): picked mon.noname-a con 0x7f95e40b4390 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:18.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.142+0000 7f95eb77c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95e41418a0 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:18.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.142+0000 7f95eb77c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e4141ce0 0x7f95e40b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:18.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.142+0000 7f95eb77c640 10 monclient(hunting): picked mon.noname-b con 0x7f95e4141ce0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:18.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.142+0000 7f95eb77c640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e40a8f60 con 0x7f95e41418a0 2023-10-30T12:10:18.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.142+0000 7f95eb77c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:18.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.142+0000 7f95eb77c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95e4067a10 con 0x7f95e40b4390 2023-10-30T12:10:18.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.142+0000 7f95eb77c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:18.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.142+0000 7f95eb77c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e40b4770 con 0x7f95e4141ce0 2023-10-30T12:10:18.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.142+0000 7f95eb77c640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:18.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.142+0000 7f95eb77c640 10 monclient(hunting): authenticate will time out at 1409.356279s 2023-10-30T12:10:18.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.142+0000 7f95e9518640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f95e40b4390 0x7f95e40a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:18.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e9d19640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e4141ce0 0x7f95e40b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:18.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e8d17640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f95e41418a0 0x7f95e40b1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.3:45972/0 (socket says 172.21.15.3:45972) 2023-10-30T12:10:18.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e8d17640 1 -- 172.21.15.3:0/1785098615 learned_addr learned my addr 172.21.15.3:0/1785098615 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:18.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e9518640 10 monclient(hunting): get_auth_request con 0x7f95e40b4390 auth_method 0 2023-10-30T12:10:18.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e9518640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:18.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e9518640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:18.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e9518640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:18.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e9d19640 10 monclient(hunting): get_auth_request con 0x7f95e4141ce0 auth_method 0 2023-10-30T12:10:18.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e9d19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:18.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e9d19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:18.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e9d19640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:18.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e9518640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:18.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e9518640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:18.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e9518640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:18.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e9d19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:18.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e9d19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:18.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.143+0000 7f95e9d19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:18.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.145+0000 7f95dbfff640 1 -- 172.21.15.3:0/1785098615 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066164308 0 0) 0x7f95d4001110 con 0x7f95e41418a0 2023-10-30T12:10:18.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.145+0000 7f95dbfff640 10 monclient(hunting): _negotiate 2023-10-30T12:10:18.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.145+0000 7f95dbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:18.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.145+0000 7f95dbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.145+0000 7f95dbfff640 10 monclient(hunting): my global_id is 4267 2023-10-30T12:10:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.145+0000 7f95dbfff640 1 -- 172.21.15.3:0/1785098615 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d0003520 con 0x7f95e41418a0 2023-10-30T12:10:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.145+0000 7f95e9518640 10 monclient(hunting): handle_auth_done global_id 4302 payload 274 2023-10-30T12:10:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.145+0000 7f95e9518640 1 -- 172.21.15.3:0/1785098615 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e4141ce0 msgr2=0x7f95e40b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.145+0000 7f95e9518640 1 --2- 172.21.15.3:0/1785098615 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e4141ce0 0x7f95e40b2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.145+0000 7f95e9518640 1 -- 172.21.15.3:0/1785098615 >> v1:172.21.15.92:6789/0 conn(0x7f95e41418a0 legacy=0x7f95e40b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.145+0000 7f95e9518640 10 monclient: _finish_hunting 0 2023-10-30T12:10:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.145+0000 7f95e9518640 1 monclient: found mon.noname-a 2023-10-30T12:10:18.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.145+0000 7f95e9518640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:18.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.145+0000 7f95e9518640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:18.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.145+0000 7f95e9518640 1 -- 172.21.15.3:0/1785098615 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e40aa3d0 con 0x7f95e40b4390 2023-10-30T12:10:18.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.147+0000 7f95e9518640 1 --2- 172.21.15.3:0/1785098615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95e40b4390 0x7f95e40a8770 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f95e00097a0 tx=0x7f95e002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=841ae90ff50f309 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:18.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95dbfff640 1 -- 172.21.15.3:0/1785098615 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e0031830 con 0x7f95e40b4390 2023-10-30T12:10:18.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:18.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95dbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:18.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95dbfff640 10 monclient: dump: 2023-10-30T12:10:18.168 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:18.168 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:18.168 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.168 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.169 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:18.169 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:18.169 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:18.169 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:18.169 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:18.170 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:18.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95dbfff640 10 monclient: _finish_auth 0 2023-10-30T12:10:18.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95dbfff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:18.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:18.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95dbfff640 1 -- 172.21.15.3:0/1785098615 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e002d040 con 0x7f95e40b4390 2023-10-30T12:10:18.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95eb77c640 5 monclient: authenticate success, global_id 4302 2023-10-30T12:10:18.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95eb77c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:18.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95dbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:18.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95eb77c640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:18.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95eb77c640 4 set_mon_vals no callback set 2023-10-30T12:10:18.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95dbfff640 1 -- 172.21.15.3:0/1785098615 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e0031bc0 con 0x7f95e40b4390 2023-10-30T12:10:18.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:18.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:18.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.151+0000 7f95dbfff640 10 monclient: dump: 2023-10-30T12:10:18.173 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:18.173 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:18.173 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.173 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.174 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:18.174 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:18.174 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:18.174 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:18.174 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:18.175 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:18.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.152+0000 7f95eb77c640 10 monclient: shutdown 2023-10-30T12:10:18.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.152+0000 7f95eb77c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:18.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.152+0000 7f95eb77c640 1 -- 172.21.15.3:0/1785098615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95e40b4390 msgr2=0x7f95e40a8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:18.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.152+0000 7f95eb77c640 1 --2- 172.21.15.3:0/1785098615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95e40b4390 0x7f95e40a8770 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f95e00097a0 tx=0x7f95e002b8d0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.152+0000 7f95eb77c640 1 -- 172.21.15.3:0/1785098615 shutdown_connections 2023-10-30T12:10:18.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.152+0000 7f95eb77c640 1 --2- 172.21.15.3:0/1785098615 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e4141ce0 0x7f95e40b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.152+0000 7f95eb77c640 1 --2- 172.21.15.3:0/1785098615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95e40b4390 0x7f95e40a8770 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.152+0000 7f95eb77c640 1 -- 172.21.15.3:0/1785098615 >> 172.21.15.3:0/1785098615 conn(0x7f95e4065ea0 msgr2=0x7f95e40662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:18.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.152+0000 7f95eb77c640 1 -- 172.21.15.3:0/1785098615 shutdown_connections 2023-10-30T12:10:18.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.152+0000 7f95eb77c640 1 -- 172.21.15.3:0/1785098615 wait complete. 2023-10-30T12:10:18.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.153+0000 7f95eb77c640 10 monclient: build_initial_monmap 2023-10-30T12:10:18.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.153+0000 7f95eb77c640 10 monclient: monmap: 2023-10-30T12:10:18.178 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:18.178 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:18.178 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:18.178 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:18.179 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:18.179 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:18.179 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:18.179 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:18.179 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:18.179 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:18.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.153+0000 7f95eb77c640 1 librados: starting msgr at 2023-10-30T12:10:18.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.153+0000 7f95eb77c640 1 librados: starting objecter 2023-10-30T12:10:18.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.153+0000 7f95eb77c640 1 Processor -- start 2023-10-30T12:10:18.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.153+0000 7f95eb77c640 1 -- start start 2023-10-30T12:10:18.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.153+0000 7f95eb77c640 1 librados: setting wanted keys 2023-10-30T12:10:18.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.153+0000 7f95eb77c640 1 librados: calling monclient init 2023-10-30T12:10:18.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.153+0000 7f95eb77c640 10 monclient: init 2023-10-30T12:10:18.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:18.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:18.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b4390 0x7f95e41da270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:18.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 10 monclient(hunting): picked mon.noname-b con 0x7f95e40b4390 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:18.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95e41418a0 0x7f95e41daab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:18.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 10 monclient(hunting): picked mon.noname-a con 0x7f95e41418a0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:18.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e4141ce0 0x7f95e413fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:18.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 10 monclient(hunting): picked mon.noname-c con 0x7f95e4141ce0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:18.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:18.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e4142470 con 0x7f95e41418a0 2023-10-30T12:10:18.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:18.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e4142f70 con 0x7f95e40b4390 2023-10-30T12:10:18.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:18.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e41430f0 con 0x7f95e4141ce0 2023-10-30T12:10:18.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:18.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95eb77c640 10 monclient(hunting): authenticate will time out at 1409.368213s 2023-10-30T12:10:18.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95e8d17640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95e41418a0 0x7f95e41daab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:18.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95e9d19640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e4141ce0 0x7f95e413fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:18.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.154+0000 7f95e9518640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b4390 0x7f95e41da270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:18.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e8d17640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95e41418a0 0x7f95e41daab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42176/0 (socket says 172.21.15.3:42176) 2023-10-30T12:10:18.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e8d17640 1 -- 172.21.15.3:0/4016152209 learned_addr learned my addr 172.21.15.3:0/4016152209 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:18.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e8d17640 10 monclient(hunting): get_auth_request con 0x7f95e41418a0 auth_method 0 2023-10-30T12:10:18.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e8d17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:18.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e8d17640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:18.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e8d17640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:18.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e9518640 10 monclient(hunting): get_auth_request con 0x7f95e40b4390 auth_method 0 2023-10-30T12:10:18.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e9518640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:18.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e9518640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:18.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e9518640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:18.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e9d19640 10 monclient(hunting): get_auth_request con 0x7f95e4141ce0 auth_method 0 2023-10-30T12:10:18.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e9d19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:18.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e9d19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:18.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e9d19640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:18.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e8d17640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:18.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e8d17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:18.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e8d17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:18.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e9518640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:18.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e9518640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:18.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e9518640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:18.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e9d19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:18.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e9d19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:18.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.155+0000 7f95e9d19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:18.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.158+0000 7f95e9518640 10 monclient(hunting): handle_auth_done global_id 4270 payload 824 2023-10-30T12:10:18.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.158+0000 7f95e9518640 1 -- 172.21.15.3:0/4016152209 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e4141ce0 msgr2=0x7f95e413fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:18.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.158+0000 7f95e9518640 1 --2- 172.21.15.3:0/4016152209 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e4141ce0 0x7f95e413fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.158+0000 7f95e9518640 1 -- 172.21.15.3:0/4016152209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95e41418a0 msgr2=0x7f95e41daab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:18.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.158+0000 7f95e9518640 1 --2- 172.21.15.3:0/4016152209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95e41418a0 0x7f95e41daab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.158+0000 7f95e9518640 10 monclient: _finish_hunting 0 2023-10-30T12:10:18.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.158+0000 7f95e9518640 1 monclient: found mon.noname-b 2023-10-30T12:10:18.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.158+0000 7f95e9518640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:18.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.158+0000 7f95e9518640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:18.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.158+0000 7f95e9518640 1 -- 172.21.15.3:0/4016152209 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e41402d0 con 0x7f95e40b4390 2023-10-30T12:10:18.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.161+0000 7f95e9518640 1 --2- 172.21.15.3:0/4016152209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b4390 0x7f95e41da270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f95e40b4cb0 tx=0x7f95e00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:18.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.161+0000 7f95d9ffb640 10 client.?.objecter ms_handle_connect 0x7f95e40b4390 2023-10-30T12:10:18.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.161+0000 7f95d9ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:18.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95d9ffb640 1 -- 172.21.15.3:0/4016152209 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e003a8f0 con 0x7f95e40b4390 2023-10-30T12:10:18.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:18.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95d9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:18.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95d9ffb640 10 monclient: dump: 2023-10-30T12:10:18.194 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:18.194 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:18.194 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.194 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.195 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:18.195 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:18.195 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:18.195 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:18.196 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:18.196 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:18.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95d9ffb640 10 monclient: _finish_auth 0 2023-10-30T12:10:18.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95d9ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:10:18.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:18.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95d9ffb640 1 -- 172.21.15.3:0/4016152209 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e003f070 con 0x7f95e40b4390 2023-10-30T12:10:18.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95d9ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:18.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95d9ffb640 1 -- 172.21.15.3:0/4016152209 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e003acc0 con 0x7f95e40b4390 2023-10-30T12:10:18.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95eb77c640 5 monclient: authenticate success, global_id 4270 2023-10-30T12:10:18.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95eb77c640 10 monclient: _renew_subs 2023-10-30T12:10:18.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95eb77c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:18.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95eb77c640 1 -- 172.21.15.3:0/4016152209 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e41450e0 con 0x7f95e40b4390 2023-10-30T12:10:18.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95db7fe640 4 set_mon_vals no callback set 2023-10-30T12:10:18.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:18.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:18.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95d9ffb640 10 monclient: dump: 2023-10-30T12:10:18.199 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:18.199 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:18.199 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.200 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.200 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:18.200 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:18.200 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:18.200 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:18.201 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:18.201 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:18.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95eb77c640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:18.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95eb77c640 10 monclient: _renew_subs 2023-10-30T12:10:18.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95eb77c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:18.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95eb77c640 1 -- 172.21.15.3:0/4016152209 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e41362c0 con 0x7f95e40b4390 2023-10-30T12:10:18.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.165+0000 7f95eb77c640 1 librados: init done 2023-10-30T12:10:18.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.166+0000 7f95eb77c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:18.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.166+0000 7f95eb77c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:18.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.166+0000 7f95eb77c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:18.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.166+0000 7f95eb77c640 1 -- 172.21.15.3:0/4016152209 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e40a9900 con 0x7f95e40b4390 2023-10-30T12:10:18.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.428+0000 7f95d9ffb640 1 -- 172.21.15.3:0/4016152209 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f95e004d440 con 0x7f95e40b4390 2023-10-30T12:10:18.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.428+0000 7f95d9ffb640 1 --2- 172.21.15.3:0/4016152209 >> v2:172.21.15.92:6800/1750155735 conn(0x7f95c806c810 0x7f95c806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:18.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.428+0000 7f95e8d17640 1 --2- 172.21.15.3:0/4016152209 >> v2:172.21.15.92:6800/1750155735 conn(0x7f95c806c810 0x7f95c806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:18.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.428+0000 7f95e8d17640 10 monclient: get_auth_request con 0x7f95c806c810 auth_method 0 2023-10-30T12:10:18.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.429+0000 7f95e8d17640 1 --2- 172.21.15.3:0/4016152209 >> v2:172.21.15.92:6800/1750155735 conn(0x7f95c806c810 0x7f95c806ecd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f95e413a9f0 tx=0x7f95d4007c50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:18.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.429+0000 7f95d9ffb640 10 client.4270.objecter ms_handle_connect 0x7f95c806c810 2023-10-30T12:10:18.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.431+0000 7f95d9ffb640 1 -- 172.21.15.3:0/4016152209 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f95e0052080 con 0x7f95e40b4390 2023-10-30T12:10:18.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.431+0000 7f95d9ffb640 10 client.4270.objecter ms_dispatch 0x7f95e4000b90 osd_map(21..21 src has 1..21) v4 2023-10-30T12:10:18.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.431+0000 7f95d9ffb640 3 client.4270.objecter handle_osd_map got epochs [21,21] > 0 2023-10-30T12:10:18.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.431+0000 7f95d9ffb640 3 client.4270.objecter handle_osd_map decoding full epoch 21 2023-10-30T12:10:18.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.431+0000 7f95d9ffb640 20 client.4270.objecter dump_active .. 0 homeless 2023-10-30T12:10:18.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.442+0000 7f95d9ffb640 1 -- 172.21.15.3:0/4016152209 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f95e003ae30 con 0x7f95e40b4390 2023-10-30T12:10:18.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.442+0000 7f95d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:18.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.442+0000 7f95d9ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:18.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.683+0000 7f95eb77c640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-10-30T12:10:18.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.683+0000 7f95eb77c640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-10-30T12:10:18.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.683+0000 7f95eb77c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:18.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.683+0000 7f95eb77c640 1 -- 172.21.15.3:0/4016152209 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f95e40a93d0 con 0x7f95e40b4390 2023-10-30T12:10:18.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.815+0000 7f95d9ffb640 1 -- 172.21.15.3:0/4016152209 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170969 (secure 0 0 0) 0x7f95e0079e60 con 0x7f95e40b4390 2023-10-30T12:10:18.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.815+0000 7f95d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-10-30T12:10:18.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.815+0000 7f95d9ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:18.820 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:10:18.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 10 librados: watch_flush enter 2023-10-30T12:10:18.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 10 librados: watch_flush exit 2023-10-30T12:10:18.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 20 client.4270.objecter shutdown clearing up homeless session... 2023-10-30T12:10:18.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 10 client.4270.objecter successfully canceled tick 2023-10-30T12:10:18.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 1 -- 172.21.15.3:0/4016152209 >> v2:172.21.15.92:6800/1750155735 conn(0x7f95c806c810 msgr2=0x7f95c806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:18.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 1 --2- 172.21.15.3:0/4016152209 >> v2:172.21.15.92:6800/1750155735 conn(0x7f95c806c810 0x7f95c806ecd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f95e413a9f0 tx=0x7f95d4007c50 comp rx=0 tx=0).stop 2023-10-30T12:10:18.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 10 monclient: shutdown 2023-10-30T12:10:18.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:18.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 1 -- 172.21.15.3:0/4016152209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b4390 msgr2=0x7f95e41da270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:18.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 1 --2- 172.21.15.3:0/4016152209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b4390 0x7f95e41da270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f95e40b4cb0 tx=0x7f95e00046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 1 -- 172.21.15.3:0/4016152209 shutdown_connections 2023-10-30T12:10:18.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 1 --2- 172.21.15.3:0/4016152209 >> v2:172.21.15.92:6800/1750155735 conn(0x7f95c806c810 0x7f95c806ecd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 1 --2- 172.21.15.3:0/4016152209 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e4141ce0 0x7f95e413fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 1 --2- 172.21.15.3:0/4016152209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95e41418a0 0x7f95e41daab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 1 --2- 172.21.15.3:0/4016152209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b4390 0x7f95e41da270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 1 -- 172.21.15.3:0/4016152209 >> 172.21.15.3:0/4016152209 conn(0x7f95e4065ea0 msgr2=0x7f95e4066280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:18.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 1 -- 172.21.15.3:0/4016152209 shutdown_connections 2023-10-30T12:10:18.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 1 -- 172.21.15.3:0/4016152209 wait complete. 2023-10-30T12:10:18.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.818+0000 7f95eb77c640 1 librados: shutdown 2023-10-30T12:10:18.831 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":5,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1750155735}]},"active_addr":"172.21.15.92:6800/1750155735","active_change":"2023-10-30T12:09:23.508893+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4105,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":1710007552}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":1866969410}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":2432702363}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":962477219}]}]} 2023-10-30T12:10:18.837 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-30T12:10:18.837 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-30T12:10:18.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:10:18.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.944+0000 7f246b145640 10 monclient: get_monmap_and_config 2023-10-30T12:10:18.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.944+0000 7f246b145640 10 monclient: build_initial_monmap 2023-10-30T12:10:18.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.944+0000 7f246b145640 10 monclient: monmap: 2023-10-30T12:10:18.946 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:18.946 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:18.946 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:18.945083+0000 2023-10-30T12:10:18.946 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:18.945083+0000 2023-10-30T12:10:18.947 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:18.947 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:18.947 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:18.947 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:18.947 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:18.948 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:18.948 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:18.948 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:18.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.944+0000 7f246b145640 1 Processor -- start 2023-10-30T12:10:18.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.944+0000 7f246b145640 1 -- start start 2023-10-30T12:10:18.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.944+0000 7f246b145640 10 monclient: init 2023-10-30T12:10:18.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.945+0000 7f246b145640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:18.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.945+0000 7f246b145640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2023-10-30T12:10:18.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.945+0000 7f246b145640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24640b4390 0x7f24640a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:18.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.945+0000 7f246b145640 10 monclient(hunting): picked mon.noname-b con 0x7f24640b4390 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:18.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.945+0000 7f246b145640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f24641418a0 0x7f2464141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:18.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.945+0000 7f246b145640 10 monclient(hunting): picked mon.noname-a con 0x7f24641418a0 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:18.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.945+0000 7f246b145640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24640b1b00 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:18.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.945+0000 7f246b145640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24640a8f60 con 0x7f24640b1b00 2023-10-30T12:10:18.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.945+0000 7f246b145640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:18.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.945+0000 7f246b145640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2464067a10 con 0x7f24641418a0 2023-10-30T12:10:18.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.945+0000 7f246b145640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:18.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.945+0000 7f246b145640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24640b4770 con 0x7f24640b4390 2023-10-30T12:10:18.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.945+0000 7f246b145640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:18.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.945+0000 7f246b145640 10 monclient(hunting): authenticate will time out at 1410.159708s 2023-10-30T12:10:18.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.946+0000 7f2463fff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f24641418a0 0x7f2464141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:18.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.946+0000 7f2463fff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f24641418a0 0x7f2464141ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42190/0 (socket says 172.21.15.3:42190) 2023-10-30T12:10:18.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.946+0000 7f2463fff640 1 -- 172.21.15.3:0/1563652433 learned_addr learned my addr 172.21.15.3:0/1563652433 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:18.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.946+0000 7f2463fff640 10 monclient(hunting): get_auth_request con 0x7f24641418a0 auth_method 0 2023-10-30T12:10:18.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.946+0000 7f2463fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:18.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.946+0000 7f2463fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:18.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.946+0000 7f2463fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:18.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.947+0000 7f2468ee1640 1 --2- 172.21.15.3:0/1563652433 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24640b4390 0x7f24640a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:18.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.947+0000 7f2463fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:18.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.947+0000 7f2463fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:18.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.947+0000 7f2463fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:18.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.948+0000 7f2468ee1640 10 monclient(hunting): get_auth_request con 0x7f24640b4390 auth_method 0 2023-10-30T12:10:18.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.948+0000 7f2468ee1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:18.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.948+0000 7f2468ee1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:18.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.948+0000 7f2468ee1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:18.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.949+0000 7f2463fff640 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2023-10-30T12:10:18.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.949+0000 7f2463fff640 1 -- 172.21.15.3:0/1563652433 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24640b4390 msgr2=0x7f24640a8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:18.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.949+0000 7f2463fff640 1 --2- 172.21.15.3:0/1563652433 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24640b4390 0x7f24640a8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.949+0000 7f2463fff640 1 -- 172.21.15.3:0/1563652433 >> v1:172.21.15.3:6789/0 conn(0x7f24640b1b00 legacy=0x7f24640b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:18.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.949+0000 7f2463fff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:18.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.949+0000 7f2463fff640 1 monclient: found mon.noname-a 2023-10-30T12:10:18.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.949+0000 7f2463fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:18.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.949+0000 7f2463fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:18.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.949+0000 7f2463fff640 1 -- 172.21.15.3:0/1563652433 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24640aa3d0 con 0x7f24641418a0 2023-10-30T12:10:18.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.953+0000 7f2463fff640 1 --2- 172.21.15.3:0/1563652433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24641418a0 0x7f2464141ce0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2450009870 tx=0x7f245002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b891e92904802b5d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:18.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.956+0000 7f24637fe640 1 -- 172.21.15.3:0/1563652433 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2450030840 con 0x7f24641418a0 2023-10-30T12:10:18.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f24637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:18.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f24637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:18.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f24637fe640 10 monclient: dump: 2023-10-30T12:10:18.961 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:18.962 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:18.962 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.962 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.962 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:18.963 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:18.963 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:18.963 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:18.963 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:18.963 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:18.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f24637fe640 10 monclient: _finish_auth 0 2023-10-30T12:10:18.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f24637fe640 10 monclient: _check_auth_tickets 2023-10-30T12:10:18.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f24637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:18.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f24637fe640 1 -- 172.21.15.3:0/1563652433 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24500309b0 con 0x7f24641418a0 2023-10-30T12:10:18.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f246b145640 5 monclient: authenticate success, global_id 4308 2023-10-30T12:10:18.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f246b145640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:18.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f24637fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:18.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f246b145640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:18.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f24637fe640 1 -- 172.21.15.3:0/1563652433 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2450030d50 con 0x7f24641418a0 2023-10-30T12:10:18.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f24637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:18.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f246b145640 4 set_mon_vals no callback set 2023-10-30T12:10:18.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f24637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:18.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f24637fe640 10 monclient: dump: 2023-10-30T12:10:18.966 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:18.966 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:18.966 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.966 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.967 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:18.967 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:18.967 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:18.967 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:18.967 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:18.968 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:18.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f246b145640 10 monclient: shutdown 2023-10-30T12:10:18.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f246b145640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:18.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f246b145640 1 -- 172.21.15.3:0/1563652433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24641418a0 msgr2=0x7f2464141ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:18.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f246b145640 1 --2- 172.21.15.3:0/1563652433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24641418a0 0x7f2464141ce0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2450009870 tx=0x7f245002b980 comp rx=0 tx=0).stop 2023-10-30T12:10:18.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f246b145640 1 -- 172.21.15.3:0/1563652433 shutdown_connections 2023-10-30T12:10:18.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f246b145640 1 --2- 172.21.15.3:0/1563652433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24641418a0 0x7f2464141ce0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f246b145640 1 --2- 172.21.15.3:0/1563652433 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24640b4390 0x7f24640a8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f246b145640 1 -- 172.21.15.3:0/1563652433 >> 172.21.15.3:0/1563652433 conn(0x7f2464065ea0 msgr2=0x7f24640662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:18.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f246b145640 1 -- 172.21.15.3:0/1563652433 shutdown_connections 2023-10-30T12:10:18.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.957+0000 7f246b145640 1 -- 172.21.15.3:0/1563652433 wait complete. 2023-10-30T12:10:18.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.958+0000 7f246b145640 10 monclient: build_initial_monmap 2023-10-30T12:10:18.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.958+0000 7f246b145640 10 monclient: monmap: 2023-10-30T12:10:18.970 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:18.970 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:18.970 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:18.971 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:18.971 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:18.971 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:18.971 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:18.971 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:18.972 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:18.972 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:18.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 1 librados: starting msgr at 2023-10-30T12:10:18.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 1 librados: starting objecter 2023-10-30T12:10:18.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 1 Processor -- start 2023-10-30T12:10:18.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 1 -- start start 2023-10-30T12:10:18.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 1 librados: setting wanted keys 2023-10-30T12:10:18.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 1 librados: calling monclient init 2023-10-30T12:10:18.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 10 monclient: init 2023-10-30T12:10:18.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:18.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:10:18.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24640b1b00 0x7f24640b64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:18.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 10 monclient(hunting): picked mon.noname-a con 0x7f24640b1b00 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:18.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24640b4390 0x7f2464145180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:18.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 10 monclient(hunting): picked mon.noname-b con 0x7f24640b4390 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:18.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24641418a0 0x7f24640b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:18.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 10 monclient(hunting): picked mon.noname-c con 0x7f24641418a0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:18.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:18.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.959+0000 7f246b145640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24640b1370 con 0x7f24640b1b00 2023-10-30T12:10:18.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.960+0000 7f246b145640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:18.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.960+0000 7f246b145640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2464145a90 con 0x7f24640b4390 2023-10-30T12:10:18.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.960+0000 7f246b145640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:18.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.960+0000 7f246b145640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24641456c0 con 0x7f24641418a0 2023-10-30T12:10:18.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.960+0000 7f246b145640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:18.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.960+0000 7f246b145640 10 monclient(hunting): authenticate will time out at 1410.174079s 2023-10-30T12:10:18.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.960+0000 7f2468ee1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24640b1b00 0x7f24640b64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:18.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.961+0000 7f2468ee1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24640b1b00 0x7f24640b64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42204/0 (socket says 172.21.15.3:42204) 2023-10-30T12:10:18.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.961+0000 7f2468ee1640 1 -- 172.21.15.3:0/2466976296 learned_addr learned my addr 172.21.15.3:0/2466976296 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:18.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.961+0000 7f2468ee1640 10 monclient(hunting): get_auth_request con 0x7f24640b1b00 auth_method 0 2023-10-30T12:10:18.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.961+0000 7f2468ee1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:18.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.961+0000 7f2468ee1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:18.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.961+0000 7f2468ee1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:18.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.961+0000 7f2468ee1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:18.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.961+0000 7f2468ee1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:18.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.961+0000 7f2468ee1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:18.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.964+0000 7f2468ee1640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2023-10-30T12:10:18.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.964+0000 7f2468ee1640 1 -- 172.21.15.3:0/2466976296 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24641418a0 msgr2=0x7f24640b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:18.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.964+0000 7f2468ee1640 1 --2- 172.21.15.3:0/2466976296 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24641418a0 0x7f24640b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.964+0000 7f2468ee1640 1 -- 172.21.15.3:0/2466976296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24640b4390 msgr2=0x7f2464145180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:18.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.964+0000 7f2468ee1640 1 --2- 172.21.15.3:0/2466976296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24640b4390 0x7f2464145180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:18.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.964+0000 7f2468ee1640 10 monclient: _finish_hunting 0 2023-10-30T12:10:18.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.964+0000 7f2468ee1640 1 monclient: found mon.noname-a 2023-10-30T12:10:18.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.964+0000 7f2468ee1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:18.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.964+0000 7f2468ee1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:18.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.964+0000 7f2468ee1640 1 -- 172.21.15.3:0/2466976296 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2464144dd0 con 0x7f24640b1b00 2023-10-30T12:10:18.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.966+0000 7f2468ee1640 1 --2- 172.21.15.3:0/2466976296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24640b1b00 0x7f24640b64e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f245800d800 tx=0x7f245800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:18.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.966+0000 7f24617fa640 10 client.?.objecter ms_handle_connect 0x7f24640b1b00 2023-10-30T12:10:18.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.966+0000 7f24617fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:18.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f24617fa640 1 -- 172.21.15.3:0/2466976296 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245800b6c0 con 0x7f24640b1b00 2023-10-30T12:10:18.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f24617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:18.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f24617fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:18.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f24617fa640 10 monclient: dump: 2023-10-30T12:10:18.984 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:18.984 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:18.984 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.984 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.984 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:18.985 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:18.985 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:18.985 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:18.985 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:18.985 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:18.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f24617fa640 10 monclient: _finish_auth 0 2023-10-30T12:10:18.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f24617fa640 10 monclient: _check_auth_tickets 2023-10-30T12:10:18.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f24617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:18.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f24617fa640 1 -- 172.21.15.3:0/2466976296 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245800b830 con 0x7f24640b1b00 2023-10-30T12:10:18.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f246b145640 5 monclient: authenticate success, global_id 4311 2023-10-30T12:10:18.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f246b145640 10 monclient: _renew_subs 2023-10-30T12:10:18.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f246b145640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:18.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f246b145640 1 -- 172.21.15.3:0/2466976296 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24640b0b20 con 0x7f24640b1b00 2023-10-30T12:10:18.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f24617fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:18.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f24617fa640 1 -- 172.21.15.3:0/2466976296 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245800bbb0 con 0x7f24640b1b00 2023-10-30T12:10:18.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f24617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:18.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f24617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:18.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f24617fa640 10 monclient: dump: 2023-10-30T12:10:18.988 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:18.988 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:18.988 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.988 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:18.988 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:18.989 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:18.989 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:18.989 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:18.989 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:18.989 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:18.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f2462ffd640 4 set_mon_vals no callback set 2023-10-30T12:10:18.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f246b145640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:18.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f246b145640 10 monclient: _renew_subs 2023-10-30T12:10:18.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f246b145640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:18.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f246b145640 1 -- 172.21.15.3:0/2466976296 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24641d9f30 con 0x7f24640b1b00 2023-10-30T12:10:18.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.970+0000 7f246b145640 1 librados: init done 2023-10-30T12:10:18.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.971+0000 7f246b145640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:18.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.971+0000 7f246b145640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:18.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.971+0000 7f246b145640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:18.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.971+0000 7f246b145640 1 -- 172.21.15.3:0/2466976296 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2464144fd0 con 0x7f24640b1b00 2023-10-30T12:10:18.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.991+0000 7f24617fa640 1 -- 172.21.15.3:0/2466976296 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f245800bd20 con 0x7f24640b1b00 2023-10-30T12:10:18.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.991+0000 7f24617fa640 1 --2- 172.21.15.3:0/2466976296 >> v2:172.21.15.92:6800/1750155735 conn(0x7f244806c810 0x7f244806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:18.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.991+0000 7f2463fff640 1 --2- 172.21.15.3:0/2466976296 >> v2:172.21.15.92:6800/1750155735 conn(0x7f244806c810 0x7f244806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:18.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.991+0000 7f2463fff640 10 monclient: get_auth_request con 0x7f244806c810 auth_method 0 2023-10-30T12:10:18.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.992+0000 7f2463fff640 1 --2- 172.21.15.3:0/2466976296 >> v2:172.21.15.92:6800/1750155735 conn(0x7f244806c810 0x7f244806ecd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2450036000 tx=0x7f245003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:18.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:18.992+0000 7f24617fa640 10 client.4311.objecter ms_handle_connect 0x7f244806c810 2023-10-30T12:10:19.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.250+0000 7f24617fa640 1 -- 172.21.15.3:0/2466976296 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f2458059640 con 0x7f24640b1b00 2023-10-30T12:10:19.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.250+0000 7f24617fa640 10 client.4311.objecter ms_dispatch 0x7f2464000b90 osd_map(21..21 src has 1..21) v4 2023-10-30T12:10:19.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.250+0000 7f24617fa640 3 client.4311.objecter handle_osd_map got epochs [21,21] > 0 2023-10-30T12:10:19.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.250+0000 7f24617fa640 3 client.4311.objecter handle_osd_map decoding full epoch 21 2023-10-30T12:10:19.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.250+0000 7f24617fa640 20 client.4311.objecter dump_active .. 0 homeless 2023-10-30T12:10:19.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.250+0000 7f24617fa640 1 -- 172.21.15.3:0/2466976296 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2458019070 con 0x7f24640b1b00 2023-10-30T12:10:19.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.250+0000 7f24617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:19.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.250+0000 7f24617fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:19.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.444+0000 7f246b145640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:10:19.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.444+0000 7f246b145640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:10:19.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.444+0000 7f246b145640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:19.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.444+0000 7f246b145640 1 -- 172.21.15.3:0/2466976296 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f246413c490 con 0x7f24640b1b00 2023-10-30T12:10:19.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.476+0000 7f24617fa640 1 -- 172.21.15.3:0/2466976296 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+12674 (secure 0 0 0) 0x7f2458058f40 con 0x7f24640b1b00 2023-10-30T12:10:19.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.476+0000 7f24617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:10:19.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.476+0000 7f24617fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:19.478 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:10:19.478 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":22,"fsid":"9bae44ae-7379-4ab2-a454-4d550607110c","created":"2023-10-30T12:09:17.680340+0000","modified":"2023-10-30T12:10:18.932264+0000","last_up_change":"2023-10-30T12:10:08.115093+0000","last_in_change":"2023-10-30T12:09:24.800567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:10:06.065827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T12:10:14.411118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"69005df2-28b6-4bb2-b3e6-79df0d403c70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3639283465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6809","nonce":3639283465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6811","nonce":3639283465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3639283465}]},"public_addr":"172.21.15.3:6808/3639283465","cluster_addr":"172.21.15.3:6809/3639283465","heartbeat_back_addr":"172.21.15.3:6811/3639283465","heartbeat_front_addr":"172.21.15.3:6810/3639283465","state":["exists","up"]},{"osd":1,"uuid":"6466ea5c-d74a-4709-a83a-f5d4fe17157e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":1445650280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6805","nonce":1445650280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6807","nonce":1445650280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":1445650280}]},"public_addr":"172.21.15.3:6804/1445650280","cluster_addr":"172.21.15.3:6805/1445650280","heartbeat_back_addr":"172.21.15.3:6807/1445650280","heartbeat_front_addr":"172.21.15.3:6806/1445650280","state":["exists","up"]},{"osd":2,"uuid":"49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":1407765917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6801","nonce":1407765917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6803","nonce":1407765917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":1407765917}]},"public_addr":"172.21.15.3:6800/1407765917","cluster_addr":"172.21.15.3:6801/1407765917","heartbeat_back_addr":"172.21.15.3:6803/1407765917","heartbeat_front_addr":"172.21.15.3:6802/1407765917","state":["exists","up"]},{"osd":3,"uuid":"69ec5657-9972-4f4b-8012-83886eb47429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2654726311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":2654726311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6815","nonce":2654726311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2654726311}]},"public_addr":"172.21.15.3:6812/2654726311","cluster_addr":"172.21.15.3:6813/2654726311","heartbeat_back_addr":"172.21.15.3:6815/2654726311","heartbeat_front_addr":"172.21.15.3:6814/2654726311","state":["exists","up"]},{"osd":4,"uuid":"da3dad24-4aeb-47a3-ad1d-d13c77cafad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6801","nonce":4229938171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4229938171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4229938171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6803","nonce":4229938171}]},"public_addr":"172.21.15.92:6801/4229938171","cluster_addr":"172.21.15.92:6802/4229938171","heartbeat_back_addr":"172.21.15.92:6804/4229938171","heartbeat_front_addr":"172.21.15.92:6803/4229938171","state":["exists","up"]},{"osd":5,"uuid":"d38d9d7f-b0de-48ab-a850-499c6a0a4364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6805","nonce":3208162957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3208162957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3208162957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6807","nonce":3208162957}]},"public_addr":"172.21.15.92:6805/3208162957","cluster_addr":"172.21.15.92:6806/3208162957","heartbeat_back_addr":"172.21.15.92:6808/3208162957","heartbeat_front_addr":"172.21.15.92:6807/3208162957","state":["exists","up"]},{"osd":6,"uuid":"b39d2b0a-84ef-4909-bac7-47f146305a79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":3275842310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3275842310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3275842310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6811","nonce":3275842310}]},"public_addr":"172.21.15.92:6809/3275842310","cluster_addr":"172.21.15.92:6810/3275842310","heartbeat_back_addr":"172.21.15.92:6812/3275842310","heartbeat_front_addr":"172.21.15.92:6811/3275842310","state":["exists","up"]},{"osd":7,"uuid":"0a9d81ac-5a14-4fef-877a-982594787dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":1101825516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1101825516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1101825516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":1101825516}]},"public_addr":"172.21.15.92:6813/1101825516","cluster_addr":"172.21.15.92:6814/1101825516","heartbeat_back_addr":"172.21.15.92:6816/1101825516","heartbeat_front_addr":"172.21.15.92:6815/1101825516","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.340529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.109770+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.885514+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.651007+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:02.744160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:03.156206+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:03.018599+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:03.334416+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:10:19.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.477+0000 7f246b145640 10 librados: watch_flush enter 2023-10-30T12:10:19.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.477+0000 7f246b145640 10 librados: watch_flush exit 2023-10-30T12:10:19.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.477+0000 7f246b145640 20 client.4311.objecter shutdown clearing up homeless session... 2023-10-30T12:10:19.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.477+0000 7f246b145640 10 client.4311.objecter successfully canceled tick 2023-10-30T12:10:19.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.478+0000 7f246b145640 1 -- 172.21.15.3:0/2466976296 >> v2:172.21.15.92:6800/1750155735 conn(0x7f244806c810 msgr2=0x7f244806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.478+0000 7f246b145640 1 --2- 172.21.15.3:0/2466976296 >> v2:172.21.15.92:6800/1750155735 conn(0x7f244806c810 0x7f244806ecd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2450036000 tx=0x7f245003b040 comp rx=0 tx=0).stop 2023-10-30T12:10:19.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.478+0000 7f246b145640 10 monclient: shutdown 2023-10-30T12:10:19.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.478+0000 7f246b145640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:19.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.478+0000 7f246b145640 1 -- 172.21.15.3:0/2466976296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24640b1b00 msgr2=0x7f24640b64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.478+0000 7f246b145640 1 --2- 172.21.15.3:0/2466976296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24640b1b00 0x7f24640b64e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f245800d800 tx=0x7f245800dcc0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.478+0000 7f246b145640 1 -- 172.21.15.3:0/2466976296 shutdown_connections 2023-10-30T12:10:19.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.478+0000 7f246b145640 1 --2- 172.21.15.3:0/2466976296 >> v2:172.21.15.92:6800/1750155735 conn(0x7f244806c810 0x7f244806ecd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.478+0000 7f246b145640 1 --2- 172.21.15.3:0/2466976296 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24641418a0 0x7f24640b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.478+0000 7f246b145640 1 --2- 172.21.15.3:0/2466976296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24640b4390 0x7f2464145180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.478+0000 7f246b145640 1 --2- 172.21.15.3:0/2466976296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24640b1b00 0x7f24640b64e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.478+0000 7f246b145640 1 -- 172.21.15.3:0/2466976296 >> 172.21.15.3:0/2466976296 conn(0x7f2464065ea0 msgr2=0x7f246413e590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:19.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.478+0000 7f246b145640 1 -- 172.21.15.3:0/2466976296 shutdown_connections 2023-10-30T12:10:19.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.478+0000 7f246b145640 1 -- 172.21.15.3:0/2466976296 wait complete. 2023-10-30T12:10:19.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.478+0000 7f246b145640 1 librados: shutdown 2023-10-30T12:10:19.490 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-30T12:10:19.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:10:19.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.599+0000 7fd6b62bb640 10 monclient: get_monmap_and_config 2023-10-30T12:10:19.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.599+0000 7fd6b62bb640 10 monclient: build_initial_monmap 2023-10-30T12:10:19.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.599+0000 7fd6b62bb640 10 monclient: monmap: 2023-10-30T12:10:19.601 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:19.601 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:19.601 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:19.600347+0000 2023-10-30T12:10:19.602 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:19.600347+0000 2023-10-30T12:10:19.602 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:19.602 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:19.602 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:19.602 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:19.603 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:19.603 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:19.603 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:19.603 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:19.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 1 Processor -- start 2023-10-30T12:10:19.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 1 -- start start 2023-10-30T12:10:19.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 10 monclient: init 2023-10-30T12:10:19.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:19.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-10-30T12:10:19.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd6b00b4390 0x7fd6b00a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 10 monclient(hunting): picked mon.noname-b con 0x7fd6b00b4390 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:19.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd6b01418a0 0x7fd6b0141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 10 monclient(hunting): picked mon.noname-c con 0x7fd6b01418a0 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:19.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6b00b1b00 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:19.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b00a8f60 con 0x7fd6b00b1b00 2023-10-30T12:10:19.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6b0067a10 con 0x7fd6b01418a0 2023-10-30T12:10:19.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b00b4770 con 0x7fd6b00b4390 2023-10-30T12:10:19.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:19.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.600+0000 7fd6b62bb640 10 monclient(hunting): authenticate will time out at 1410.814906s 2023-10-30T12:10:19.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.638+0000 7fd6af7fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd6b01418a0 0x7fd6b0141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.639+0000 7fd6af7fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd6b01418a0 0x7fd6b0141ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56204/0 (socket says 172.21.15.3:56204) 2023-10-30T12:10:19.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.639+0000 7fd6af7fe640 1 -- 172.21.15.3:0/2019517068 learned_addr learned my addr 172.21.15.3:0/2019517068 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:19.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.639+0000 7fd6af7fe640 10 monclient(hunting): get_auth_request con 0x7fd6b01418a0 auth_method 0 2023-10-30T12:10:19.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.639+0000 7fd6af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.639+0000 7fd6af7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.639+0000 7fd6af7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.640+0000 7fd6af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.640+0000 7fd6af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.640+0000 7fd6af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.641+0000 7fd6af7fe640 10 monclient(hunting): handle_auth_done global_id 4273 payload 274 2023-10-30T12:10:19.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.641+0000 7fd6af7fe640 1 -- 172.21.15.3:0/2019517068 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd6b00b4390 msgr2=0x7fd6b00a8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:19.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.641+0000 7fd6af7fe640 1 --2- 172.21.15.3:0/2019517068 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd6b00b4390 0x7fd6b00a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.641+0000 7fd6af7fe640 1 -- 172.21.15.3:0/2019517068 >> v1:172.21.15.92:6789/0 conn(0x7fd6b00b1b00 legacy=0x7fd6b00b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.641+0000 7fd6af7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:10:19.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.641+0000 7fd6af7fe640 1 monclient: found mon.noname-c 2023-10-30T12:10:19.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.641+0000 7fd6af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:19.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.641+0000 7fd6af7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:10:19.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.641+0000 7fd6af7fe640 1 -- 172.21.15.3:0/2019517068 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b00aa3d0 con 0x7fd6b01418a0 2023-10-30T12:10:19.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.646+0000 7fd6af7fe640 1 --2- 172.21.15.3:0/2019517068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6b01418a0 0x7fd6b0141ce0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4009870 tx=0x7fd6a402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6361b76bbb55207 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6aeffd640 1 -- 172.21.15.3:0/2019517068 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4030840 con 0x7fd6b01418a0 2023-10-30T12:10:19.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6aeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:19.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6aeffd640 10 monclient: dump: 2023-10-30T12:10:19.655 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:19.655 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:19.656 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:19.656 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:19.656 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.656 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:19.656 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:19.657 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:19.657 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:19.657 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:19.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6aeffd640 10 monclient: _finish_auth 0 2023-10-30T12:10:19.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6aeffd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:19.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:19.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6aeffd640 1 -- 172.21.15.3:0/2019517068 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a40309b0 con 0x7fd6b01418a0 2023-10-30T12:10:19.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6b62bb640 5 monclient: authenticate success, global_id 4273 2023-10-30T12:10:19.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6b62bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:19.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6aeffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:19.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6b62bb640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:19.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6aeffd640 1 -- 172.21.15.3:0/2019517068 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4030d50 con 0x7fd6b01418a0 2023-10-30T12:10:19.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6b62bb640 4 set_mon_vals no callback set 2023-10-30T12:10:19.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:19.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6aeffd640 10 monclient: dump: 2023-10-30T12:10:19.660 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:19.660 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:19.660 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:19.660 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:19.660 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.661 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:19.661 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:19.661 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:19.661 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:19.661 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:19.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6b62bb640 10 monclient: shutdown 2023-10-30T12:10:19.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6b62bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:19.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6b62bb640 1 -- 172.21.15.3:0/2019517068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6b01418a0 msgr2=0x7fd6b0141ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.653+0000 7fd6b62bb640 1 --2- 172.21.15.3:0/2019517068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6b01418a0 0x7fd6b0141ce0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4009870 tx=0x7fd6a402b980 comp rx=0 tx=0).stop 2023-10-30T12:10:19.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.654+0000 7fd6b62bb640 1 -- 172.21.15.3:0/2019517068 shutdown_connections 2023-10-30T12:10:19.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.654+0000 7fd6b62bb640 1 --2- 172.21.15.3:0/2019517068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6b01418a0 0x7fd6b0141ce0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.654+0000 7fd6b62bb640 1 --2- 172.21.15.3:0/2019517068 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd6b00b4390 0x7fd6b00a8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.654+0000 7fd6b62bb640 1 -- 172.21.15.3:0/2019517068 >> 172.21.15.3:0/2019517068 conn(0x7fd6b0065ea0 msgr2=0x7fd6b00662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:19.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.654+0000 7fd6b62bb640 1 -- 172.21.15.3:0/2019517068 shutdown_connections 2023-10-30T12:10:19.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.654+0000 7fd6b62bb640 1 -- 172.21.15.3:0/2019517068 wait complete. 2023-10-30T12:10:19.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.654+0000 7fd6b62bb640 10 monclient: build_initial_monmap 2023-10-30T12:10:19.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.654+0000 7fd6b62bb640 10 monclient: monmap: 2023-10-30T12:10:19.668 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:19.668 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:19.668 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:19.668 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:19.669 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:19.669 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:19.669 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:19.669 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:19.669 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:19.670 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:19.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 1 librados: starting msgr at 2023-10-30T12:10:19.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 1 librados: starting objecter 2023-10-30T12:10:19.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 1 Processor -- start 2023-10-30T12:10:19.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 1 -- start start 2023-10-30T12:10:19.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 1 librados: setting wanted keys 2023-10-30T12:10:19.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 1 librados: calling monclient init 2023-10-30T12:10:19.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 10 monclient: init 2023-10-30T12:10:19.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:19.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:10:19.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6b00b1b00 0x7fd6b0144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 10 monclient(hunting): picked mon.noname-a con 0x7fd6b00b1b00 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:19.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6b00b4390 0x7fd6b013e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 10 monclient(hunting): picked mon.noname-b con 0x7fd6b00b4390 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:19.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd6b01418a0 0x7fd6b01d9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 10 monclient(hunting): picked mon.noname-c con 0x7fd6b01418a0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:19.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b00b13b0 con 0x7fd6b00b1b00 2023-10-30T12:10:19.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b01455c0 con 0x7fd6b00b4390 2023-10-30T12:10:19.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b013f150 con 0x7fd6b01418a0 2023-10-30T12:10:19.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:19.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.655+0000 7fd6b62bb640 10 monclient(hunting): authenticate will time out at 1410.870006s 2023-10-30T12:10:19.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.656+0000 7fd6affff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6b00b1b00 0x7fd6b0144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.656+0000 7fd6b4858640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd6b01418a0 0x7fd6b01d9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.656+0000 7fd6af7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6b00b4390 0x7fd6b013e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.656+0000 7fd6affff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6b00b1b00 0x7fd6b0144220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42208/0 (socket says 172.21.15.3:42208) 2023-10-30T12:10:19.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.656+0000 7fd6affff640 1 -- 172.21.15.3:0/2766937353 learned_addr learned my addr 172.21.15.3:0/2766937353 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:19.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.656+0000 7fd6b4858640 10 monclient(hunting): get_auth_request con 0x7fd6b01418a0 auth_method 0 2023-10-30T12:10:19.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.656+0000 7fd6b4858640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.656+0000 7fd6b4858640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.656+0000 7fd6b4858640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.656+0000 7fd6affff640 10 monclient(hunting): get_auth_request con 0x7fd6b00b1b00 auth_method 0 2023-10-30T12:10:19.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.656+0000 7fd6affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.656+0000 7fd6affff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.656+0000 7fd6affff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.657+0000 7fd6af7fe640 10 monclient(hunting): get_auth_request con 0x7fd6b00b4390 auth_method 0 2023-10-30T12:10:19.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.657+0000 7fd6af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.657+0000 7fd6af7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.657+0000 7fd6af7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.657+0000 7fd6affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.657+0000 7fd6affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.657+0000 7fd6affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.657+0000 7fd6b4858640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.657+0000 7fd6b4858640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.657+0000 7fd6b4858640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.657+0000 7fd6af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.657+0000 7fd6af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.657+0000 7fd6af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.660+0000 7fd6af7fe640 10 monclient(hunting): handle_auth_done global_id 4276 payload 824 2023-10-30T12:10:19.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.660+0000 7fd6af7fe640 1 -- 172.21.15.3:0/2766937353 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd6b01418a0 msgr2=0x7fd6b01d9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.660+0000 7fd6af7fe640 1 --2- 172.21.15.3:0/2766937353 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd6b01418a0 0x7fd6b01d9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.660+0000 7fd6af7fe640 1 -- 172.21.15.3:0/2766937353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6b00b1b00 msgr2=0x7fd6b0144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.660+0000 7fd6af7fe640 1 --2- 172.21.15.3:0/2766937353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6b00b1b00 0x7fd6b0144220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.660+0000 7fd6af7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:10:19.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.660+0000 7fd6af7fe640 1 monclient: found mon.noname-b 2023-10-30T12:10:19.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.660+0000 7fd6af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:19.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.660+0000 7fd6af7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:19.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.660+0000 7fd6af7fe640 1 -- 172.21.15.3:0/2766937353 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b0144cc0 con 0x7fd6b00b4390 2023-10-30T12:10:19.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.662+0000 7fd6af7fe640 1 --2- 172.21.15.3:0/2766937353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6b00b4390 0x7fd6b013e500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4033000 tx=0x7fd6a4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.662+0000 7fd6acff9640 10 client.?.objecter ms_handle_connect 0x7fd6b00b4390 2023-10-30T12:10:19.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.662+0000 7fd6acff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:19.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.665+0000 7fd6acff9640 1 -- 172.21.15.3:0/2766937353 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4039d90 con 0x7fd6b00b4390 2023-10-30T12:10:19.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.665+0000 7fd6acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.665+0000 7fd6acff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:19.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.665+0000 7fd6acff9640 10 monclient: dump: 2023-10-30T12:10:19.694 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:19.695 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:19.695 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:19.695 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:19.696 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.696 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:19.696 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:19.697 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:19.697 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:19.697 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:19.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.665+0000 7fd6acff9640 10 monclient: _finish_auth 0 2023-10-30T12:10:19.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.665+0000 7fd6acff9640 10 monclient: _check_auth_tickets 2023-10-30T12:10:19.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.665+0000 7fd6acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:19.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.665+0000 7fd6acff9640 1 -- 172.21.15.3:0/2766937353 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a40433f0 con 0x7fd6b00b4390 2023-10-30T12:10:19.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.665+0000 7fd6acff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:19.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.665+0000 7fd6b62bb640 5 monclient: authenticate success, global_id 4276 2023-10-30T12:10:19.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.665+0000 7fd6acff9640 1 -- 172.21.15.3:0/2766937353 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a40437b0 con 0x7fd6b00b4390 2023-10-30T12:10:19.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.666+0000 7fd6acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.666+0000 7fd6acff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:19.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.666+0000 7fd6acff9640 10 monclient: dump: 2023-10-30T12:10:19.702 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:19.702 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:19.703 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:19.703 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:19.703 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.703 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:19.704 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:19.704 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:19.704 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:19.704 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:19.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.666+0000 7fd6b62bb640 10 monclient: _renew_subs 2023-10-30T12:10:19.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.666+0000 7fd6ae7fc640 4 set_mon_vals no callback set 2023-10-30T12:10:19.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.666+0000 7fd6b62bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:19.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.666+0000 7fd6b62bb640 1 -- 172.21.15.3:0/2766937353 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b01da510 con 0x7fd6b00b4390 2023-10-30T12:10:19.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.666+0000 7fd6b62bb640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:19.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.666+0000 7fd6b62bb640 10 monclient: _renew_subs 2023-10-30T12:10:19.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.666+0000 7fd6b62bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:19.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.666+0000 7fd6b62bb640 1 -- 172.21.15.3:0/2766937353 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b01da9b0 con 0x7fd6b00b4390 2023-10-30T12:10:19.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.666+0000 7fd6b62bb640 1 librados: init done 2023-10-30T12:10:19.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.667+0000 7fd6b62bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:19.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.667+0000 7fd6b62bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:19.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.667+0000 7fd6b62bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:19.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.667+0000 7fd6b62bb640 1 -- 172.21.15.3:0/2766937353 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b00a93d0 con 0x7fd6b00b4390 2023-10-30T12:10:19.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.685+0000 7fd6acff9640 1 -- 172.21.15.3:0/2766937353 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fd6a404c3b0 con 0x7fd6b00b4390 2023-10-30T12:10:19.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.686+0000 7fd6acff9640 1 --2- 172.21.15.3:0/2766937353 >> v2:172.21.15.92:6800/1750155735 conn(0x7fd69c06c810 0x7fd69c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.686+0000 7fd6affff640 1 --2- 172.21.15.3:0/2766937353 >> v2:172.21.15.92:6800/1750155735 conn(0x7fd69c06c810 0x7fd69c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.686+0000 7fd6affff640 10 monclient: get_auth_request con 0x7fd69c06c810 auth_method 0 2023-10-30T12:10:19.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.686+0000 7fd6affff640 1 --2- 172.21.15.3:0/2766937353 >> v2:172.21.15.92:6800/1750155735 conn(0x7fd69c06c810 0x7fd69c06ecd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009a40 tx=0x7fd6a00091c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.686+0000 7fd6acff9640 10 client.4276.objecter ms_handle_connect 0x7fd69c06c810 2023-10-30T12:10:19.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.907+0000 7fd6acff9640 1 -- 172.21.15.3:0/2766937353 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fd6a407a030 con 0x7fd6b00b4390 2023-10-30T12:10:19.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.907+0000 7fd6acff9640 10 client.4276.objecter ms_dispatch 0x7fd6b0000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:19.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.907+0000 7fd6acff9640 3 client.4276.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:19.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.907+0000 7fd6acff9640 3 client.4276.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:19.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.907+0000 7fd6acff9640 20 client.4276.objecter dump_active .. 0 homeless 2023-10-30T12:10:19.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.917+0000 7fd6acff9640 1 -- 172.21.15.3:0/2766937353 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd6a407e0d0 con 0x7fd6b00b4390 2023-10-30T12:10:19.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.917+0000 7fd6acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:19.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:19.917+0000 7fd6acff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:20.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.098+0000 7fd6b62bb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:10:20.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.098+0000 7fd6b62bb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:10:20.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.098+0000 7fd6b62bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.098+0000 7fd6b62bb640 1 -- 172.21.15.3:0/2766937353 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6b01da510 con 0x7fd6b00b4390 2023-10-30T12:10:20.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.189+0000 7fd6acff9640 1 -- 172.21.15.3:0/2766937353 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+12674 (secure 0 0 0) 0x7fd6a4082280 con 0x7fd6b00b4390 2023-10-30T12:10:20.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.189+0000 7fd6acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:10:20.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.189+0000 7fd6acff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:20.192 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:10:20.192 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":22,"fsid":"9bae44ae-7379-4ab2-a454-4d550607110c","created":"2023-10-30T12:09:17.680340+0000","modified":"2023-10-30T12:10:18.932264+0000","last_up_change":"2023-10-30T12:10:08.115093+0000","last_in_change":"2023-10-30T12:09:24.800567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:10:06.065827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T12:10:14.411118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"69005df2-28b6-4bb2-b3e6-79df0d403c70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3639283465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6809","nonce":3639283465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6811","nonce":3639283465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3639283465}]},"public_addr":"172.21.15.3:6808/3639283465","cluster_addr":"172.21.15.3:6809/3639283465","heartbeat_back_addr":"172.21.15.3:6811/3639283465","heartbeat_front_addr":"172.21.15.3:6810/3639283465","state":["exists","up"]},{"osd":1,"uuid":"6466ea5c-d74a-4709-a83a-f5d4fe17157e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":1445650280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6805","nonce":1445650280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6807","nonce":1445650280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":1445650280}]},"public_addr":"172.21.15.3:6804/1445650280","cluster_addr":"172.21.15.3:6805/1445650280","heartbeat_back_addr":"172.21.15.3:6807/1445650280","heartbeat_front_addr":"172.21.15.3:6806/1445650280","state":["exists","up"]},{"osd":2,"uuid":"49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":1407765917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6801","nonce":1407765917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6803","nonce":1407765917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":1407765917}]},"public_addr":"172.21.15.3:6800/1407765917","cluster_addr":"172.21.15.3:6801/1407765917","heartbeat_back_addr":"172.21.15.3:6803/1407765917","heartbeat_front_addr":"172.21.15.3:6802/1407765917","state":["exists","up"]},{"osd":3,"uuid":"69ec5657-9972-4f4b-8012-83886eb47429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2654726311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":2654726311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6815","nonce":2654726311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2654726311}]},"public_addr":"172.21.15.3:6812/2654726311","cluster_addr":"172.21.15.3:6813/2654726311","heartbeat_back_addr":"172.21.15.3:6815/2654726311","heartbeat_front_addr":"172.21.15.3:6814/2654726311","state":["exists","up"]},{"osd":4,"uuid":"da3dad24-4aeb-47a3-ad1d-d13c77cafad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6801","nonce":4229938171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4229938171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4229938171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6803","nonce":4229938171}]},"public_addr":"172.21.15.92:6801/4229938171","cluster_addr":"172.21.15.92:6802/4229938171","heartbeat_back_addr":"172.21.15.92:6804/4229938171","heartbeat_front_addr":"172.21.15.92:6803/4229938171","state":["exists","up"]},{"osd":5,"uuid":"d38d9d7f-b0de-48ab-a850-499c6a0a4364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6805","nonce":3208162957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3208162957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3208162957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6807","nonce":3208162957}]},"public_addr":"172.21.15.92:6805/3208162957","cluster_addr":"172.21.15.92:6806/3208162957","heartbeat_back_addr":"172.21.15.92:6808/3208162957","heartbeat_front_addr":"172.21.15.92:6807/3208162957","state":["exists","up"]},{"osd":6,"uuid":"b39d2b0a-84ef-4909-bac7-47f146305a79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":3275842310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3275842310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3275842310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6811","nonce":3275842310}]},"public_addr":"172.21.15.92:6809/3275842310","cluster_addr":"172.21.15.92:6810/3275842310","heartbeat_back_addr":"172.21.15.92:6812/3275842310","heartbeat_front_addr":"172.21.15.92:6811/3275842310","state":["exists","up"]},{"osd":7,"uuid":"0a9d81ac-5a14-4fef-877a-982594787dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":1101825516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1101825516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1101825516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":1101825516}]},"public_addr":"172.21.15.92:6813/1101825516","cluster_addr":"172.21.15.92:6814/1101825516","heartbeat_back_addr":"172.21.15.92:6816/1101825516","heartbeat_front_addr":"172.21.15.92:6815/1101825516","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.340529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.109770+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.885514+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.651007+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:02.744160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:03.156206+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:03.018599+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:03.334416+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:10:20.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 10 librados: watch_flush enter 2023-10-30T12:10:20.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 10 librados: watch_flush exit 2023-10-30T12:10:20.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 20 client.4276.objecter shutdown clearing up homeless session... 2023-10-30T12:10:20.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 10 client.4276.objecter successfully canceled tick 2023-10-30T12:10:20.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 1 -- 172.21.15.3:0/2766937353 >> v2:172.21.15.92:6800/1750155735 conn(0x7fd69c06c810 msgr2=0x7fd69c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 1 --2- 172.21.15.3:0/2766937353 >> v2:172.21.15.92:6800/1750155735 conn(0x7fd69c06c810 0x7fd69c06ecd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009a40 tx=0x7fd6a00091c0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 10 monclient: shutdown 2023-10-30T12:10:20.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 1 -- 172.21.15.3:0/2766937353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6b00b4390 msgr2=0x7fd6b013e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 1 --2- 172.21.15.3:0/2766937353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6b00b4390 0x7fd6b013e500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4033000 tx=0x7fd6a4004730 comp rx=0 tx=0).stop 2023-10-30T12:10:20.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 1 -- 172.21.15.3:0/2766937353 shutdown_connections 2023-10-30T12:10:20.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 1 --2- 172.21.15.3:0/2766937353 >> v2:172.21.15.92:6800/1750155735 conn(0x7fd69c06c810 0x7fd69c06ecd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 1 --2- 172.21.15.3:0/2766937353 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd6b01418a0 0x7fd6b01d9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 1 --2- 172.21.15.3:0/2766937353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6b00b4390 0x7fd6b013e500 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 1 --2- 172.21.15.3:0/2766937353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6b00b1b00 0x7fd6b0144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 1 -- 172.21.15.3:0/2766937353 >> 172.21.15.3:0/2766937353 conn(0x7fd6b0065ea0 msgr2=0x7fd6b01399f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 1 -- 172.21.15.3:0/2766937353 shutdown_connections 2023-10-30T12:10:20.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 1 -- 172.21.15.3:0/2766937353 wait complete. 2023-10-30T12:10:20.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.191+0000 7fd6b62bb640 1 librados: shutdown 2023-10-30T12:10:20.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-30T12:10:20.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-30T12:10:20.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-30T12:10:20.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-30T12:10:20.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-30T12:10:20.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-30T12:10:20.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-30T12:10:20.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-30T12:10:20.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.399+0000 7fecb1f16640 10 monclient: get_monmap_and_config 2023-10-30T12:10:20.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.399+0000 7fecb1f16640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.399+0000 7fecb1f16640 10 monclient: monmap: 2023-10-30T12:10:20.407 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.407 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.407 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:20.400533+0000 2023-10-30T12:10:20.407 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:20.400533+0000 2023-10-30T12:10:20.408 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.408 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.408 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:20.408 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:20.408 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:20.409 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:20.409 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:20.409 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.401+0000 7f06ac9c2640 10 monclient: get_monmap_and_config 2023-10-30T12:10:20.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.401+0000 7f06ac9c2640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.401+0000 7f06ac9c2640 10 monclient: monmap: 2023-10-30T12:10:20.411 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.411 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.412 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:20.402054+0000 2023-10-30T12:10:20.412 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:20.402054+0000 2023-10-30T12:10:20.412 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.413 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.413 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:20.413 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:20.414 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:20.414 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:20.414 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:20.415 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.402+0000 7f06ac9c2640 1 Processor -- start 2023-10-30T12:10:20.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.402+0000 7f06ac9c2640 1 -- start start 2023-10-30T12:10:20.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.402+0000 7f06ac9c2640 10 monclient: init 2023-10-30T12:10:20.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.403+0000 7f06ac9c2640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.403+0000 7f06ac9c2640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-10-30T12:10:20.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.403+0000 7f06ac9c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06a8151df0 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:20.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.403+0000 7f06ac9c2640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f06a81521d0 0x7f06a806ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.403+0000 7f06ac9c2640 10 monclient(hunting): picked mon.noname-b con 0x7f06a81521d0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.403+0000 7f06ac9c2640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f06a8151580 0x7f06a80ae690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.403+0000 7f06ac9c2640 10 monclient(hunting): picked mon.noname-a con 0x7f06a8151580 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:20.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.403+0000 7f06ac9c2640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a806e2c0 con 0x7f06a8151df0 2023-10-30T12:10:20.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.403+0000 7f06ac9c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.403+0000 7f06ac9c2640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a8067b90 con 0x7f06a8151580 2023-10-30T12:10:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.403+0000 7f06ac9c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.403+0000 7f06ac9c2640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a806fdc0 con 0x7f06a81521d0 2023-10-30T12:10:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.403+0000 7f06ac9c2640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.403+0000 7f06ac9c2640 10 monclient(hunting): authenticate will time out at 1411.617443s 2023-10-30T12:10:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.405+0000 7f06a659c640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f06a8151df0 0x7f06a8150e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.3:45992/0 (socket says 172.21.15.3:45992) 2023-10-30T12:10:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.405+0000 7f06a659c640 1 -- 172.21.15.3:0/981438618 learned_addr learned my addr 172.21.15.3:0/981438618 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.405+0000 7f06a5d9b640 1 --2- 172.21.15.3:0/981438618 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f06a81521d0 0x7f06a806ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.405+0000 7f06a6d9d640 1 --2- 172.21.15.3:0/981438618 >> v2:172.21.15.3:3300/0 conn(0x7f06a8151580 0x7f06a80ae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.405+0000 7fecb1f16640 1 Processor -- start 2023-10-30T12:10:20.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.405+0000 7fecb1f16640 1 -- start start 2023-10-30T12:10:20.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.405+0000 7fecb1f16640 10 monclient: init 2023-10-30T12:10:20.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7f06a6d9d640 10 monclient(hunting): get_auth_request con 0x7f06a8151580 auth_method 0 2023-10-30T12:10:20.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7f06a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7f06a6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7f06a6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-10-30T12:10:20.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecac0ae6b0 0x7fecac0aac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 10 monclient(hunting): picked mon.noname-b con 0x7fecac0ae6b0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fecac0ab1d0 0x7fecac0a9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 10 monclient(hunting): picked mon.noname-a con 0x7fecac0ab1d0 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:20.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fecac0aa0c0 0x7fecac0aa510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 10 monclient(hunting): picked mon.noname-c con 0x7fecac0aa0c0 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:20.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecac067a30 con 0x7fecac0ab1d0 2023-10-30T12:10:20.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecac0ac1a0 con 0x7fecac0aa0c0 2023-10-30T12:10:20.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecac0aea90 con 0x7fecac0ae6b0 2023-10-30T12:10:20.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.406+0000 7fecb1f16640 10 monclient(hunting): authenticate will time out at 1411.620778s 2023-10-30T12:10:20.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.407+0000 7fecaaffd640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fecac0ab1d0 0x7fecac0a9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.407+0000 7fecabfff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fecac0aa0c0 0x7fecac0aa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.407+0000 7fecaaffd640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fecac0ab1d0 0x7fecac0a9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42228/0 (socket says 172.21.15.3:42228) 2023-10-30T12:10:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.407+0000 7fecaaffd640 1 -- 172.21.15.3:0/4052798722 learned_addr learned my addr 172.21.15.3:0/4052798722 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.407+0000 7fecaaffd640 10 monclient(hunting): get_auth_request con 0x7fecac0ab1d0 auth_method 0 2023-10-30T12:10:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.407+0000 7fecaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.407+0000 7fecaaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.407+0000 7fecaaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.407+0000 7f06a5d9b640 10 monclient(hunting): get_auth_request con 0x7f06a81521d0 auth_method 0 2023-10-30T12:10:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.407+0000 7f06a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.407+0000 7f06a5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.407+0000 7f06a5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.408+0000 7fecabfff640 10 monclient(hunting): get_auth_request con 0x7fecac0aa0c0 auth_method 0 2023-10-30T12:10:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.408+0000 7fecabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.408+0000 7fecabfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.408+0000 7fecabfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.408+0000 7fe472f47640 10 monclient: get_monmap_and_config 2023-10-30T12:10:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.408+0000 7fe472f47640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.408+0000 7fe472f47640 10 monclient: monmap: 2023-10-30T12:10:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:20.409845+0000 2023-10-30T12:10:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:20.409845+0000 2023-10-30T12:10:20.435 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.435 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.435 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:20.435 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:20.436 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:20.436 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:20.436 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:20.436 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.409+0000 7f06a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.409+0000 7f06a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.409+0000 7f06a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.409+0000 7fecaaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.409+0000 7fecaaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.409+0000 7fecaaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7fecab7fe640 1 --2- 172.21.15.3:0/4052798722 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecac0ae6b0 0x7fecac0aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7fecab7fe640 10 monclient(hunting): get_auth_request con 0x7fecac0ae6b0 auth_method 0 2023-10-30T12:10:20.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7fecab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7fecab7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7fecab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7f06a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7f06a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7f06a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7f2c1ca50640 10 monclient: get_monmap_and_config 2023-10-30T12:10:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7f2c1ca50640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7f2c1ca50640 10 monclient: monmap: 2023-10-30T12:10:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.443 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.443 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:20.411033+0000 2023-10-30T12:10:20.443 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:20.411033+0000 2023-10-30T12:10:20.443 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.444 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.444 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:20.444 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:20.444 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:20.445 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:20.445 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:20.445 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.411+0000 7f2c1ca50640 1 Processor -- start 2023-10-30T12:10:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7f06a559a640 1 -- 172.21.15.3:0/981438618 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264798453 0 0) 0x7f069c004850 con 0x7f06a8151df0 2023-10-30T12:10:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7f06a559a640 10 monclient(hunting): _negotiate 2023-10-30T12:10:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7f06a559a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7f06a559a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7f06a559a640 10 monclient(hunting): my global_id is 4279 2023-10-30T12:10:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7f06a559a640 1 -- 172.21.15.3:0/981438618 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0690003520 con 0x7f06a8151df0 2023-10-30T12:10:20.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7fecabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7fecabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7fecabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.411+0000 7f2c1ca50640 1 -- start start 2023-10-30T12:10:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.411+0000 7f2c1ca50640 10 monclient: init 2023-10-30T12:10:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7f2c1ca50640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7f2c1ca50640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2023-10-30T12:10:20.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fecab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fecab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fecab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7f2c1ca50640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c180ab220 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:20.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7f2c1ca50640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2c180ae530 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:20.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7f2c1ca50640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2c180ae970 0x7f2c181518c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.410+0000 7fe472f47640 1 Processor -- start 2023-10-30T12:10:20.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.411+0000 7fe472f47640 1 -- start start 2023-10-30T12:10:20.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.411+0000 7fe472f47640 10 monclient: init 2023-10-30T12:10:20.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fe472f47640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fe472f47640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-10-30T12:10:20.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fe472f47640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe46c0ae7a0 0x7fe46c0a95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fe472f47640 10 monclient(hunting): picked mon.noname-c con 0x7fe46c0ae7a0 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fe472f47640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe46c149810 0x7fe46c149c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fe472f47640 10 monclient(hunting): picked mon.noname-b con 0x7fe46c149810 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fe472f47640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe46c0a9c70 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fe472f47640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe46c0ace60 con 0x7fe46c0a9c70 2023-10-30T12:10:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fe472f47640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fe472f47640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe46c067a30 con 0x7fe46c0ae7a0 2023-10-30T12:10:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fe472f47640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fe472f47640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c0aeb80 con 0x7fe46c149810 2023-10-30T12:10:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fe472f47640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7fe472f47640 10 monclient(hunting): authenticate will time out at 1411.626284s 2023-10-30T12:10:20.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7f2c1ca50640 10 monclient(hunting): picked mon.noname-b con 0x7f2c180ae970 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7f2c1ca50640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1814a150 con 0x7f2c180ae530 2023-10-30T12:10:20.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7f2c1ca50640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c180acc00 con 0x7f2c180ab220 2023-10-30T12:10:20.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7f2c1ca50640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7f2c1ca50640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18067a30 con 0x7f2c180ae970 2023-10-30T12:10:20.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7f2c1ca50640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.412+0000 7f2c1ca50640 10 monclient(hunting): authenticate will time out at 1411.626570s 2023-10-30T12:10:20.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7f3699ffb640 10 monclient: get_monmap_and_config 2023-10-30T12:10:20.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7f3699ffb640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7f3699ffb640 10 monclient: monmap: 2023-10-30T12:10:20.462 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.463 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.463 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:20.415798+0000 2023-10-30T12:10:20.463 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:20.415798+0000 2023-10-30T12:10:20.463 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.463 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.464 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:20.464 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:20.464 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:20.464 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:20.465 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:20.465 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7f06a5d9b640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2023-10-30T12:10:20.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7f06a5d9b640 1 -- 172.21.15.3:0/981438618 >> v2:172.21.15.3:3300/0 conn(0x7f06a8151580 msgr2=0x7f06a80ae690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:20.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7f06a5d9b640 1 --2- 172.21.15.3:0/981438618 >> v2:172.21.15.3:3300/0 conn(0x7f06a8151580 0x7f06a80ae690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7f06a5d9b640 1 -- 172.21.15.3:0/981438618 >> v1:172.21.15.92:6789/0 conn(0x7f06a8151df0 legacy=0x7f06a8150e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7f06a5d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7f06a5d9b640 1 monclient: found mon.noname-b 2023-10-30T12:10:20.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7f06a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7f06a5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7f06a5d9b640 1 -- 172.21.15.3:0/981438618 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a80700a0 con 0x7f06a81521d0 2023-10-30T12:10:20.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7fecabfff640 10 monclient(hunting): handle_auth_done global_id 4282 payload 274 2023-10-30T12:10:20.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7fecabfff640 1 -- 172.21.15.3:0/4052798722 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecac0ae6b0 msgr2=0x7fecac0aac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:20.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7fecabfff640 1 --2- 172.21.15.3:0/4052798722 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecac0ae6b0 0x7fecac0aac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7fecabfff640 1 -- 172.21.15.3:0/4052798722 >> v2:172.21.15.3:3300/0 conn(0x7fecac0ab1d0 msgr2=0x7fecac0a9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:20.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7fecabfff640 1 --2- 172.21.15.3:0/4052798722 >> v2:172.21.15.3:3300/0 conn(0x7fecac0ab1d0 0x7fecac0a9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7fecabfff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7fecabfff640 1 monclient: found mon.noname-c 2023-10-30T12:10:20.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7fecabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7fecabfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.414+0000 7fecabfff640 1 -- 172.21.15.3:0/4052798722 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac0abec0 con 0x7fecac0aa0c0 2023-10-30T12:10:20.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.415+0000 7f3699ffb640 1 Processor -- start 2023-10-30T12:10:20.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.415+0000 7f3699ffb640 1 -- start start 2023-10-30T12:10:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.415+0000 7f3699ffb640 10 monclient: init 2023-10-30T12:10:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.416+0000 7f3699ffb640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.416+0000 7f3699ffb640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-10-30T12:10:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.416+0000 7f3699ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f36940a88d0 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.417+0000 7f3699ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f36940b1b00 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:20.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.417+0000 7f3699ffb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3694143930 0x7f36940a8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.417+0000 7f3699ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f3694143930 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.417+0000 7f3699ffb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36940a91f0 con 0x7f36940a88d0 2023-10-30T12:10:20.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.417+0000 7f3699ffb640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36940aa250 con 0x7f36940b1b00 2023-10-30T12:10:20.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.417+0000 7f3699ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.417+0000 7f3699ffb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36940679a0 con 0x7f3694143930 2023-10-30T12:10:20.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.417+0000 7f3699ffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.417+0000 7f3699ffb640 10 monclient(hunting): authenticate will time out at 1411.631338s 2023-10-30T12:10:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.419+0000 7fe2bdd65640 10 monclient: get_monmap_and_config 2023-10-30T12:10:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.419+0000 7fe2bdd65640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.419+0000 7fe2bdd65640 10 monclient: monmap: 2023-10-30T12:10:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:20.420261+0000 2023-10-30T12:10:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:20.420261+0000 2023-10-30T12:10:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.476 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:20.476 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:20.476 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:20.476 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:20.476 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:20.477 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.419+0000 7f777ce2d640 10 monclient: get_monmap_and_config 2023-10-30T12:10:20.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.419+0000 7f777ce2d640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.419+0000 7f777ce2d640 10 monclient: monmap: 2023-10-30T12:10:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:20.420506+0000 2023-10-30T12:10:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:20.420506+0000 2023-10-30T12:10:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:20.480 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.420+0000 7fe2bdd65640 1 Processor -- start 2023-10-30T12:10:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.420+0000 7fe2bdd65640 1 -- start start 2023-10-30T12:10:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.420+0000 7fe2bdd65640 10 monclient: init 2023-10-30T12:10:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.420+0000 7f777ce2d640 1 Processor -- start 2023-10-30T12:10:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.420+0000 7f777ce2d640 1 -- start start 2023-10-30T12:10:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.420+0000 7f777ce2d640 10 monclient: init 2023-10-30T12:10:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.420+0000 7fe471f45640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe46c0ae7a0 0x7fe46c0a95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.420+0000 7fe2bdd65640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.420+0000 7fe471f45640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe46c0ae7a0 0x7fe46c0a95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56236/0 (socket says 172.21.15.3:56236) 2023-10-30T12:10:20.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.420+0000 7fe471f45640 1 -- 172.21.15.3:0/3856763079 learned_addr learned my addr 172.21.15.3:0/3856763079 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:20.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.420+0000 7fe471f45640 10 monclient(hunting): get_auth_request con 0x7fe46c0ae7a0 auth_method 0 2023-10-30T12:10:20.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.420+0000 7fe471f45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.420+0000 7fe471f45640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.420+0000 7fe471f45640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.420+0000 7fe2bdd65640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2023-10-30T12:10:20.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f777ce2d640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f777ce2d640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2023-10-30T12:10:20.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f777ce2d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f77780a8e40 0x7f777813efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f777ce2d640 10 monclient(hunting): picked mon.noname-b con 0x7f77780a8e40 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f777ce2d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f77780b13c0 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:20.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f777ce2d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f777813f500 0x7f77781425c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f777ce2d640 10 monclient(hunting): picked mon.noname-c con 0x7f777813f500 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:20.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f777ce2d640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77780b1b00 con 0x7f77780b13c0 2023-10-30T12:10:20.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f777ce2d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f777ce2d640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77780679a0 con 0x7f777813f500 2023-10-30T12:10:20.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f777ce2d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f777ce2d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77780b1060 con 0x7f77780a8e40 2023-10-30T12:10:20.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f777ce2d640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f777ce2d640 10 monclient(hunting): authenticate will time out at 1411.635267s 2023-10-30T12:10:20.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7fe2bdd65640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2b80a8e40 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:20.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7fe2bdd65640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe2b80b13c0 0x7fe2b8141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7fe2bdd65640 10 monclient(hunting): picked mon.noname-a con 0x7fe2b80b13c0 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:20.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7fe2bdd65640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2b81423f0 0x7fe2b814c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7fe2bdd65640 10 monclient(hunting): picked mon.noname-b con 0x7fe2b81423f0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7fe2bdd65640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b80b1b00 con 0x7fe2b80a8e40 2023-10-30T12:10:20.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7fe2bdd65640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7fe2bdd65640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b80679a0 con 0x7fe2b80b13c0 2023-10-30T12:10:20.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7fe2bdd65640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7fe2bdd65640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b80b1060 con 0x7fe2b81423f0 2023-10-30T12:10:20.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7fe2bdd65640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7fe2bdd65640 10 monclient(hunting): authenticate will time out at 1411.635164s 2023-10-30T12:10:20.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f3692ffd640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f36940b1b00 0x7f3694143220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.3:46020/0 (socket says 172.21.15.3:46020) 2023-10-30T12:10:20.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f3692ffd640 1 -- 172.21.15.3:0/3650660865 learned_addr learned my addr 172.21.15.3:0/3650660865 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:20.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7fe471f45640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7fe471f45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7f2c1759e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2c180ae970 0x7f2c181518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.421+0000 7fe471f45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.422+0000 7f7776d9d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f777813f500 0x7f77781425c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.422+0000 7f2c1759e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2c180ae970 0x7f2c181518c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:60708/0 (socket says 172.21.15.3:60708) 2023-10-30T12:10:20.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.422+0000 7f2c1759e640 1 -- 172.21.15.3:0/3969496969 learned_addr learned my addr 172.21.15.3:0/3969496969 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:20.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.422+0000 7f2c1759e640 10 monclient(hunting): get_auth_request con 0x7f2c180ae970 auth_method 0 2023-10-30T12:10:20.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.422+0000 7f2c1759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.422+0000 7f2c1759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.422+0000 7f2c1759e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.423+0000 7f2c1759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.423+0000 7f2c1759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.423+0000 7f2c1759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.423+0000 7fe471f45640 10 monclient(hunting): handle_auth_done global_id 4285 payload 274 2023-10-30T12:10:20.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.423+0000 7fe471f45640 1 -- 172.21.15.3:0/3856763079 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe46c149810 msgr2=0x7fe46c149c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:20.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.423+0000 7fe471f45640 1 --2- 172.21.15.3:0/3856763079 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe46c149810 0x7fe46c149c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.423+0000 7fe471f45640 1 -- 172.21.15.3:0/3856763079 >> v1:172.21.15.3:6789/0 conn(0x7fe46c0a9c70 legacy=0x7fe46c0aa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:20.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.423+0000 7fe471f45640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.423+0000 7fe471f45640 1 monclient: found mon.noname-c 2023-10-30T12:10:20.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.423+0000 7fe471f45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.423+0000 7fe471f45640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.423+0000 7fe471f45640 1 -- 172.21.15.3:0/3856763079 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe46c0abf70 con 0x7fe46c0ae7a0 2023-10-30T12:10:20.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.424+0000 7fe2b6ffd640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe2b80b13c0 0x7fe2b8141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.424+0000 7f3693fff640 1 --2- 172.21.15.3:0/3650660865 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3694143930 0x7f36940a8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.424+0000 7f7776d9d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f777813f500 0x7f77781425c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56244/0 (socket says 172.21.15.3:56244) 2023-10-30T12:10:20.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.424+0000 7f7776d9d640 1 -- 172.21.15.3:0/1053082998 learned_addr learned my addr 172.21.15.3:0/1053082998 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:20.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.424+0000 7f7776d9d640 10 monclient(hunting): get_auth_request con 0x7f777813f500 auth_method 0 2023-10-30T12:10:20.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.424+0000 7f7776d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.424+0000 7f7776d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.424+0000 7f7776d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.425+0000 7fe2b6ffd640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe2b80b13c0 0x7fe2b8141eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42234/0 (socket says 172.21.15.3:42234) 2023-10-30T12:10:20.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.425+0000 7fe2b6ffd640 1 -- 172.21.15.3:0/1791755096 learned_addr learned my addr 172.21.15.3:0/1791755096 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:20.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.425+0000 7fe2b6ffd640 10 monclient(hunting): get_auth_request con 0x7fe2b80b13c0 auth_method 0 2023-10-30T12:10:20.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.425+0000 7fe2b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.425+0000 7fe2b6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.425+0000 7fe2b6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.425+0000 7f7776d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.425+0000 7f7776d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.425+0000 7f7776d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.426+0000 7f2c1759e640 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2023-10-30T12:10:20.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.426+0000 7f2c1759e640 1 -- 172.21.15.3:0/3969496969 >> v1:172.21.15.92:6789/0 conn(0x7f2c180ab220 legacy=0x7f2c18149330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:20.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.426+0000 7f2c1759e640 1 -- 172.21.15.3:0/3969496969 >> v1:172.21.15.3:6789/0 conn(0x7f2c180ae530 legacy=0x7f2c18149a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:20.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.426+0000 7f2c1759e640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.426+0000 7f2c1759e640 1 monclient: found mon.noname-b 2023-10-30T12:10:20.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.426+0000 7f2c1759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.426+0000 7f2c1759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.426+0000 7f2c1759e640 1 -- 172.21.15.3:0/3969496969 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c180ab600 con 0x7f2c180ae970 2023-10-30T12:10:20.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.427+0000 7f3693fff640 10 monclient(hunting): get_auth_request con 0x7f3694143930 auth_method 0 2023-10-30T12:10:20.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.427+0000 7f3693fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.427+0000 7f3693fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.427+0000 7f3693fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.427+0000 7fe471f45640 1 --2- 172.21.15.3:0/3856763079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46c0ae7a0 0x7fe46c0a95e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe464009e50 tx=0x7fe46402bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17ec4517d1a79b21 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.428+0000 7fe2b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.428+0000 7fe2b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.428+0000 7fe2b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.429+0000 7fe2b7fff640 1 --2- 172.21.15.3:0/1791755096 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2b81423f0 0x7fe2b814c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.429+0000 7f7776d9d640 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2023-10-30T12:10:20.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.429+0000 7f7776d9d640 1 -- 172.21.15.3:0/1053082998 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f77780a8e40 msgr2=0x7f777813efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:20.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.429+0000 7f7776d9d640 1 --2- 172.21.15.3:0/1053082998 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f77780a8e40 0x7f777813efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.429+0000 7f7776d9d640 1 -- 172.21.15.3:0/1053082998 >> v1:172.21.15.92:6789/0 conn(0x7f77780b13c0 legacy=0x7f7778141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:20.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.429+0000 7f7776d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.429+0000 7f7776d9d640 1 monclient: found mon.noname-c 2023-10-30T12:10:20.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.429+0000 7f7776d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.429+0000 7f7776d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.429+0000 7f7776d9d640 1 -- 172.21.15.3:0/1053082998 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77780a8920 con 0x7f777813f500 2023-10-30T12:10:20.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.430+0000 7f2c1759e640 1 --2- 172.21.15.3:0/3969496969 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2c180ae970 0x7f2c181518c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009e80 tx=0x7f2c1002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2f4cebd80918008 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.430+0000 7fe2b6ffd640 10 monclient(hunting): handle_auth_done global_id 4323 payload 274 2023-10-30T12:10:20.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.430+0000 7fe2b6ffd640 1 -- 172.21.15.3:0/1791755096 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2b81423f0 msgr2=0x7fe2b814c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:20.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.430+0000 7fe2b6ffd640 1 --2- 172.21.15.3:0/1791755096 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2b81423f0 0x7fe2b814c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.430+0000 7fe2b6ffd640 1 -- 172.21.15.3:0/1791755096 >> v1:172.21.15.92:6789/0 conn(0x7fe2b80a8e40 legacy=0x7fe2b813efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:20.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.430+0000 7fe2b6ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.430+0000 7fe2b6ffd640 1 monclient: found mon.noname-a 2023-10-30T12:10:20.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.430+0000 7fe2b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.430+0000 7fe2b6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:20.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.430+0000 7fe2b6ffd640 1 -- 172.21.15.3:0/1791755096 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b80a8920 con 0x7fe2b80b13c0 2023-10-30T12:10:20.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.430+0000 7fecabfff640 1 --2- 172.21.15.3:0/4052798722 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecac0aa0c0 0x7fecac0aa510 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fecac066950 tx=0x7feca002b7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a4c937a299c68ab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.430+0000 7f601674e640 10 monclient: get_monmap_and_config 2023-10-30T12:10:20.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.430+0000 7f601674e640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.430+0000 7f601674e640 10 monclient: monmap: 2023-10-30T12:10:20.518 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.518 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.519 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:20.431943+0000 2023-10-30T12:10:20.519 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:20.431943+0000 2023-10-30T12:10:20.519 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.519 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.519 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:20.520 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:20.520 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:20.520 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:20.520 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:20.520 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.431+0000 7f3693fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.431+0000 7f3693fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.431+0000 7f3693fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.431+0000 7f601674e640 1 Processor -- start 2023-10-30T12:10:20.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.432+0000 7f601674e640 1 -- start start 2023-10-30T12:10:20.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.432+0000 7f601674e640 10 monclient: init 2023-10-30T12:10:20.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.432+0000 7f7776d9d640 1 --2- 172.21.15.3:0/1053082998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f777813f500 0x7f77781425c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7770009e50 tx=0x7f777002bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aff85638f1ead92b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.432+0000 7f601674e640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.432+0000 7f601674e640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-10-30T12:10:20.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.432+0000 7f601674e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60100a8e40 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:20.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.432+0000 7f601674e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60100b13c0 0x7f6010141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.432+0000 7f601674e640 10 monclient(hunting): picked mon.noname-b con 0x7f60100b13c0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.432+0000 7f601674e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f60101423f0 0x7f601014c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.432+0000 7f601674e640 10 monclient(hunting): picked mon.noname-c con 0x7f60101423f0 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:20.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.432+0000 7f601674e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60100b1b00 con 0x7f60100a8e40 2023-10-30T12:10:20.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.432+0000 7f601674e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.433+0000 7f601674e640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60100679a0 con 0x7f60101423f0 2023-10-30T12:10:20.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.433+0000 7f601674e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.433+0000 7f601674e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60100b1060 con 0x7f60100b13c0 2023-10-30T12:10:20.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.433+0000 7f601674e640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.433+0000 7f601674e640 10 monclient(hunting): authenticate will time out at 1411.647095s 2023-10-30T12:10:20.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.435+0000 7f36927fc640 1 -- 172.21.15.3:0/3650660865 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630228300 0 0) 0x7f3684001110 con 0x7f36940a88d0 2023-10-30T12:10:20.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.435+0000 7f36927fc640 10 monclient(hunting): _negotiate 2023-10-30T12:10:20.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.435+0000 7f36927fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.435+0000 7f36927fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.435+0000 7f36927fc640 10 monclient(hunting): my global_id is 4326 2023-10-30T12:10:20.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.435+0000 7f36927fc640 1 -- 172.21.15.3:0/3650660865 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f367c003520 con 0x7f36940a88d0 2023-10-30T12:10:20.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.435+0000 7f600ffff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f60100a8e40 0x7f601013efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50416/0 (socket says 172.21.15.3:50416) 2023-10-30T12:10:20.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.435+0000 7f600ffff640 1 -- 172.21.15.3:0/3809707824 learned_addr learned my addr 172.21.15.3:0/3809707824 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:20.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.436+0000 7fe2b6ffd640 1 --2- 172.21.15.3:0/1791755096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2b80b13c0 0x7fe2b8141eb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009870 tx=0x7fe2ac02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbec28b96b1ee93 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.436+0000 7f3693fff640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2023-10-30T12:10:20.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.436+0000 7f3693fff640 1 -- 172.21.15.3:0/3650660865 >> v1:172.21.15.92:6789/0 conn(0x7f36940b1b00 legacy=0x7f3694143220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.436+0000 7f3693fff640 1 -- 172.21.15.3:0/3650660865 >> v1:172.21.15.3:6789/0 conn(0x7f36940a88d0 legacy=0x7f3694146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.436+0000 7f3693fff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.436+0000 7f3693fff640 1 monclient: found mon.noname-b 2023-10-30T12:10:20.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.436+0000 7f3693fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.436+0000 7f3693fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.436+0000 7f3693fff640 1 -- 172.21.15.3:0/3650660865 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36940a9450 con 0x7f3694143930 2023-10-30T12:10:20.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777559a640 1 -- 172.21.15.3:0/1053082998 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777002c950 con 0x7f777813f500 2023-10-30T12:10:20.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:20.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777559a640 10 monclient: dump: 2023-10-30T12:10:20.533 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.533 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.533 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.533 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.534 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.534 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.534 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.534 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.534 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.535 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777559a640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777559a640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777559a640 1 -- 172.21.15.3:0/1053082998 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777002cac0 con 0x7f777813f500 2023-10-30T12:10:20.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777559a640 1 -- 172.21.15.3:0/1053082998 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777002ce60 con 0x7f777813f500 2023-10-30T12:10:20.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777ce2d640 5 monclient: authenticate success, global_id 4288 2023-10-30T12:10:20.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777ce2d640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:20.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777ce2d640 4 set_mon_vals no callback set 2023-10-30T12:10:20.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:20.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f777559a640 10 monclient: dump: 2023-10-30T12:10:20.537 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.538 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.538 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.538 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.538 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.538 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.539 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.539 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.539 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.539 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.439+0000 7f06a5d9b640 1 --2- 172.21.15.3:0/981438618 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f06a81521d0 0x7f06a806ecf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f06a000c2e0 tx=0x7f06a0030570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48ff022117ced203 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.440+0000 7f777ce2d640 10 monclient: shutdown 2023-10-30T12:10:20.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.440+0000 7f777ce2d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.440+0000 7f777ce2d640 1 -- 172.21.15.3:0/1053082998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f777813f500 msgr2=0x7f77781425c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.440+0000 7f777ce2d640 1 --2- 172.21.15.3:0/1053082998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f777813f500 0x7f77781425c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7770009e50 tx=0x7f777002bdb0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.440+0000 7f777ce2d640 1 -- 172.21.15.3:0/1053082998 shutdown_connections 2023-10-30T12:10:20.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.440+0000 7f777ce2d640 1 --2- 172.21.15.3:0/1053082998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f777813f500 0x7f77781425c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.440+0000 7f777ce2d640 1 --2- 172.21.15.3:0/1053082998 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f77780a8e40 0x7f777813efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.440+0000 7f777ce2d640 1 -- 172.21.15.3:0/1053082998 >> 172.21.15.3:0/1053082998 conn(0x7f77780b7180 msgr2=0x7f77780b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.440+0000 7f777ce2d640 1 -- 172.21.15.3:0/1053082998 shutdown_connections 2023-10-30T12:10:20.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.440+0000 7f777ce2d640 1 -- 172.21.15.3:0/1053082998 wait complete. 2023-10-30T12:10:20.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.441+0000 7f777ce2d640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.441+0000 7f777ce2d640 10 monclient: monmap: 2023-10-30T12:10:20.543 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.544 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:20.544 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:20.544 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:20.544 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.544 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:20.545 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:20.545 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:20.545 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:20.545 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.441+0000 7f777ce2d640 1 librados: starting msgr at 2023-10-30T12:10:20.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.441+0000 7f777ce2d640 1 librados: starting objecter 2023-10-30T12:10:20.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.441+0000 7f777ce2d640 1 Processor -- start 2023-10-30T12:10:20.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.441+0000 7f777ce2d640 1 -- start start 2023-10-30T12:10:20.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.441+0000 7f777ce2d640 1 librados: setting wanted keys 2023-10-30T12:10:20.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.441+0000 7f777ce2d640 1 librados: calling monclient init 2023-10-30T12:10:20.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.441+0000 7f777ce2d640 10 monclient: init 2023-10-30T12:10:20.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.441+0000 7fecaa7fc640 1 -- 172.21.15.3:0/4052798722 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca0033620 con 0x7fecac0aa0c0 2023-10-30T12:10:20.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.441+0000 7f6014ceb640 1 --2- 172.21.15.3:0/3809707824 >> v2:172.21.15.92:3300/0 conn(0x7f60101423f0 0x7f601014c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecaa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:20.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecaa7fc640 10 monclient: dump: 2023-10-30T12:10:20.549 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.549 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.549 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.550 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.550 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.550 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.550 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.551 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.551 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.551 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecaa7fc640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecaa7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecb1f16640 5 monclient: authenticate success, global_id 4282 2023-10-30T12:10:20.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecb1f16640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:20.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f6014ceb640 10 monclient(hunting): get_auth_request con 0x7f60101423f0 auth_method 0 2023-10-30T12:10:20.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f6014ceb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecaa7fc640 1 -- 172.21.15.3:0/4052798722 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca0033790 con 0x7fecac0aa0c0 2023-10-30T12:10:20.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecaa7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f6014ceb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f6014ceb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:20.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77780a8e40 0x7f77780ae550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 10 monclient(hunting): picked mon.noname-b con 0x7f77780a8e40 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:20.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77780abba0 0x7f77781dad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 10 monclient(hunting): picked mon.noname-a con 0x7f77780abba0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:20.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f77780ab170 0x7f77781db250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 10 monclient(hunting): picked mon.noname-c con 0x7f77780ab170 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77780a9a10 con 0x7f77780abba0 2023-10-30T12:10:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77780ac180 con 0x7f77780a8e40 2023-10-30T12:10:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77780aecb0 con 0x7f77780ab170 2023-10-30T12:10:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7f777ce2d640 10 monclient(hunting): authenticate will time out at 1411.656745s 2023-10-30T12:10:20.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecb1f16640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:20.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecb1f16640 4 set_mon_vals no callback set 2023-10-30T12:10:20.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecaa7fc640 1 -- 172.21.15.3:0/4052798722 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca0033b70 con 0x7fecac0aa0c0 2023-10-30T12:10:20.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecaa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:20.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.442+0000 7fecaa7fc640 10 monclient: dump: 2023-10-30T12:10:20.560 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.560 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.561 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.561 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.561 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.563 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7fecb1f16640 10 monclient: shutdown 2023-10-30T12:10:20.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7fecb1f16640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7fecb1f16640 1 -- 172.21.15.3:0/4052798722 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecac0aa0c0 msgr2=0x7fecac0aa510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7fecb1f16640 1 --2- 172.21.15.3:0/4052798722 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecac0aa0c0 0x7fecac0aa510 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fecac066950 tx=0x7feca002b7d0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7fecb1f16640 1 -- 172.21.15.3:0/4052798722 shutdown_connections 2023-10-30T12:10:20.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7fecb1f16640 1 --2- 172.21.15.3:0/4052798722 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecac0aa0c0 0x7fecac0aa510 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7fecb1f16640 1 --2- 172.21.15.3:0/4052798722 >> v2:172.21.15.3:3300/0 conn(0x7fecac0ab1d0 0x7fecac0a9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7fecb1f16640 1 --2- 172.21.15.3:0/4052798722 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecac0ae6b0 0x7fecac0aac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7fecb1f16640 1 -- 172.21.15.3:0/4052798722 >> 172.21.15.3:0/4052798722 conn(0x7fecac069d50 msgr2=0x7fecac06a160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7fecb1f16640 1 -- 172.21.15.3:0/4052798722 shutdown_connections 2023-10-30T12:10:20.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7fecb1f16640 1 -- 172.21.15.3:0/4052798722 wait complete. 2023-10-30T12:10:20.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7f600f7fe640 1 --2- 172.21.15.3:0/3809707824 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60100b13c0 0x7f6010141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7f6014ceb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7f6014ceb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7f6014ceb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7fecb1f16640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.443+0000 7fecb1f16640 10 monclient: monmap: 2023-10-30T12:10:20.568 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.568 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:20.568 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:20.569 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:20.569 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.569 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:20.569 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:20.569 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:20.569 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:20.570 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.444+0000 7fecb1f16640 1 librados: starting msgr at 2023-10-30T12:10:20.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.444+0000 7fecb1f16640 1 librados: starting objecter 2023-10-30T12:10:20.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.444+0000 7fecb1f16640 1 Processor -- start 2023-10-30T12:10:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.444+0000 7fecb1f16640 1 -- start start 2023-10-30T12:10:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.444+0000 7fecb1f16640 1 librados: setting wanted keys 2023-10-30T12:10:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.444+0000 7fecb1f16640 1 librados: calling monclient init 2023-10-30T12:10:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.444+0000 7fecb1f16640 10 monclient: init 2023-10-30T12:10:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.444+0000 7f600f7fe640 10 monclient(hunting): get_auth_request con 0x7f60100b13c0 auth_method 0 2023-10-30T12:10:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.444+0000 7f600f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.444+0000 7f600f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.444+0000 7f600f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:10:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7f3693fff640 1 --2- 172.21.15.3:0/3650660865 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3694143930 0x7f36940a8cb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f368c009dc0 tx=0x7f368c02be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43191f5d4050f34e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecac0aa0c0 0x7fecac1ddee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 10 monclient(hunting): picked mon.noname-b con 0x7fecac0aa0c0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecac0ab1d0 0x7fecac0a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 10 monclient(hunting): picked mon.noname-c con 0x7fecac0ab1d0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecac0ae6b0 0x7fecac0a2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 10 monclient(hunting): picked mon.noname-a con 0x7fecac0ae6b0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac0a9c10 con 0x7fecac0ae6b0 2023-10-30T12:10:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac0a26b0 con 0x7fecac0aa0c0 2023-10-30T12:10:20.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecac0a9350 con 0x7fecac0ab1d0 2023-10-30T12:10:20.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.445+0000 7fecb1f16640 10 monclient(hunting): authenticate will time out at 1411.659262s 2023-10-30T12:10:20.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.446+0000 7f600effd640 1 -- 172.21.15.3:0/3809707824 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313995365 0 0) 0x7f6008001490 con 0x7f60100a8e40 2023-10-30T12:10:20.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.446+0000 7f600effd640 10 monclient(hunting): _negotiate 2023-10-30T12:10:20.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.446+0000 7f600effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.446+0000 7f600effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.446+0000 7f600effd640 10 monclient(hunting): my global_id is 4329 2023-10-30T12:10:20.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.446+0000 7f600effd640 1 -- 172.21.15.3:0/3809707824 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ffc003520 con 0x7f60100a8e40 2023-10-30T12:10:20.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.446+0000 7f6014ceb640 10 monclient(hunting): handle_auth_done global_id 4294 payload 274 2023-10-30T12:10:20.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.446+0000 7f6014ceb640 1 -- 172.21.15.3:0/3809707824 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60100b13c0 msgr2=0x7f6010141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:20.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.446+0000 7f6014ceb640 1 --2- 172.21.15.3:0/3809707824 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60100b13c0 0x7f6010141eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.446+0000 7f6014ceb640 1 -- 172.21.15.3:0/3809707824 >> v1:172.21.15.3:6789/0 conn(0x7f60100a8e40 legacy=0x7f601013efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.446+0000 7f6014ceb640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.446+0000 7f6014ceb640 1 monclient: found mon.noname-c 2023-10-30T12:10:20.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.446+0000 7f6014ceb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.446+0000 7f6014ceb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.446+0000 7f6014ceb640 1 -- 172.21.15.3:0/3809707824 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60100a8920 con 0x7f60101423f0 2023-10-30T12:10:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7f7775d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77780abba0 0x7f77781dad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe470f43640 1 -- 172.21.15.3:0/3856763079 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46402c950 con 0x7fe46c0ae7a0 2023-10-30T12:10:20.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe470f43640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe470f43640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:20.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe470f43640 10 monclient: dump: 2023-10-30T12:10:20.586 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.587 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.587 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.587 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.588 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.588 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.590 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe470f43640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe470f43640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe470f43640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe472f47640 5 monclient: authenticate success, global_id 4285 2023-10-30T12:10:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe472f47640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:20.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe470f43640 1 -- 172.21.15.3:0/3856763079 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46402cac0 con 0x7fe46c0ae7a0 2023-10-30T12:10:20.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe470f43640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe472f47640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:20.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe472f47640 4 set_mon_vals no callback set 2023-10-30T12:10:20.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe470f43640 1 -- 172.21.15.3:0/3856763079 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46402c950 con 0x7fe46c0ae7a0 2023-10-30T12:10:20.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe470f43640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe470f43640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.448+0000 7fe470f43640 10 monclient: dump: 2023-10-30T12:10:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.595 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.595 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.595 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.596 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.596 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.596 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.597 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.597 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7fe472f47640 10 monclient: shutdown 2023-10-30T12:10:20.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7fe472f47640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7fe472f47640 1 -- 172.21.15.3:0/3856763079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46c0ae7a0 msgr2=0x7fe46c0a95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7fe472f47640 1 --2- 172.21.15.3:0/3856763079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46c0ae7a0 0x7fe46c0a95e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe464009e50 tx=0x7fe46402bdb0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7fe472f47640 1 -- 172.21.15.3:0/3856763079 shutdown_connections 2023-10-30T12:10:20.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7fe472f47640 1 --2- 172.21.15.3:0/3856763079 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe46c149810 0x7fe46c149c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7fe472f47640 1 --2- 172.21.15.3:0/3856763079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46c0ae7a0 0x7fe46c0a95e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7fe472f47640 1 -- 172.21.15.3:0/3856763079 >> 172.21.15.3:0/3856763079 conn(0x7fe46c069d50 msgr2=0x7fe46c06a130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7f7775d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77780abba0 0x7f77781dad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42248/0 (socket says 172.21.15.3:42248) 2023-10-30T12:10:20.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7f7775d9b640 1 -- 172.21.15.3:0/2982470888 learned_addr learned my addr 172.21.15.3:0/2982470888 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:20.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7f7775d9b640 10 monclient(hunting): get_auth_request con 0x7f77780abba0 auth_method 0 2023-10-30T12:10:20.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7f7775d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7f7775d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7f7775d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7fe472f47640 1 -- 172.21.15.3:0/3856763079 shutdown_connections 2023-10-30T12:10:20.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7fe472f47640 1 -- 172.21.15.3:0/3856763079 wait complete. 2023-10-30T12:10:20.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe472f47640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe472f47640 10 monclient: monmap: 2023-10-30T12:10:20.603 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.603 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:20.603 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:20.603 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:20.603 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.604 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:20.604 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:20.604 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:20.604 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:20.605 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe472f47640 1 librados: starting msgr at 2023-10-30T12:10:20.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe472f47640 1 librados: starting objecter 2023-10-30T12:10:20.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe472f47640 1 Processor -- start 2023-10-30T12:10:20.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7fe2b67fc640 1 -- 172.21.15.3:0/1791755096 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac031740 con 0x7fe2b80b13c0 2023-10-30T12:10:20.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.449+0000 7fe2b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2b67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2b67fc640 10 monclient: dump: 2023-10-30T12:10:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.609 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2b67fc640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2b67fc640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2b67fc640 1 -- 172.21.15.3:0/1791755096 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac0318b0 con 0x7fe2b80b13c0 2023-10-30T12:10:20.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2bdd65640 5 monclient: authenticate success, global_id 4323 2023-10-30T12:10:20.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2b67fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2b67fc640 1 -- 172.21.15.3:0/1791755096 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac031c50 con 0x7fe2b80b13c0 2023-10-30T12:10:20.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:20.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2b67fc640 10 monclient: dump: 2023-10-30T12:10:20.611 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.611 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.611 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.613 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2bdd65640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2bdd65640 4 set_mon_vals no callback set 2023-10-30T12:10:20.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe472f47640 1 -- start start 2023-10-30T12:10:20.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe472f47640 1 librados: setting wanted keys 2023-10-30T12:10:20.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe472f47640 1 librados: calling monclient init 2023-10-30T12:10:20.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe472f47640 10 monclient: init 2023-10-30T12:10:20.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7f7775d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7f7775d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7f7775d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2bdd65640 10 monclient: shutdown 2023-10-30T12:10:20.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2bdd65640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2bdd65640 1 -- 172.21.15.3:0/1791755096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2b80b13c0 msgr2=0x7fe2b8141eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2bdd65640 1 --2- 172.21.15.3:0/1791755096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2b80b13c0 0x7fe2b8141eb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009870 tx=0x7fe2ac02b7e0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2bdd65640 1 -- 172.21.15.3:0/1791755096 shutdown_connections 2023-10-30T12:10:20.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2bdd65640 1 --2- 172.21.15.3:0/1791755096 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2b81423f0 0x7fe2b814c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2bdd65640 1 --2- 172.21.15.3:0/1791755096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2b80b13c0 0x7fe2b8141eb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.450+0000 7fe2bdd65640 1 -- 172.21.15.3:0/1791755096 >> 172.21.15.3:0/1791755096 conn(0x7fe2b80b7180 msgr2=0x7fe2b80b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe2bdd65640 1 -- 172.21.15.3:0/1791755096 shutdown_connections 2023-10-30T12:10:20.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe2bdd65640 1 -- 172.21.15.3:0/1791755096 wait complete. 2023-10-30T12:10:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:10:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe46c0a9c70 0x7fe46c14c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 10 monclient(hunting): picked mon.noname-a con 0x7fe46c0a9c70 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe46c0ae7a0 0x7fe46c14cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 10 monclient(hunting): picked mon.noname-c con 0x7fe46c0ae7a0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46c149810 0x7fe46c14d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 10 monclient(hunting): picked mon.noname-b con 0x7fe46c149810 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:20.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c065e20 con 0x7fe46c0a9c70 2023-10-30T12:10:20.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c14f1d0 con 0x7fe46c149810 2023-10-30T12:10:20.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c14ae90 con 0x7fe46c0ae7a0 2023-10-30T12:10:20.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe472f47640 10 monclient(hunting): authenticate will time out at 1411.665350s 2023-10-30T12:10:20.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe2bdd65640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe2bdd65640 10 monclient: monmap: 2023-10-30T12:10:20.625 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.625 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:20.625 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:20.626 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:20.626 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.626 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:20.627 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:20.627 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:20.627 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:20.627 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c15d9b640 1 -- 172.21.15.3:0/3969496969 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1002ca50 con 0x7f2c180ae970 2023-10-30T12:10:20.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c15d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:20.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c15d9b640 10 monclient: dump: 2023-10-30T12:10:20.629 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.630 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.630 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.630 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.630 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.630 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.631 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.631 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.631 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.631 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c15d9b640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c15d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c15d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c15d9b640 1 -- 172.21.15.3:0/3969496969 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1002cbc0 con 0x7f2c180ae970 2023-10-30T12:10:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c15d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c15d9b640 1 -- 172.21.15.3:0/3969496969 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c100343f0 con 0x7f2c180ae970 2023-10-30T12:10:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c15d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c15d9b640 10 monclient: dump: 2023-10-30T12:10:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.634 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.634 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.634 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.634 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.635 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.635 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.635 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe2bdd65640 1 librados: starting msgr at 2023-10-30T12:10:20.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7fe2bdd65640 1 librados: starting objecter 2023-10-30T12:10:20.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c1ca50640 5 monclient: authenticate success, global_id 4235 2023-10-30T12:10:20.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c1ca50640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:20.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.451+0000 7f2c1ca50640 4 set_mon_vals no callback set 2023-10-30T12:10:20.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7fe2bdd65640 1 Processor -- start 2023-10-30T12:10:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7fe2bdd65640 1 -- start start 2023-10-30T12:10:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7fe2bdd65640 1 librados: setting wanted keys 2023-10-30T12:10:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7fe2bdd65640 1 librados: calling monclient init 2023-10-30T12:10:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7fe2bdd65640 10 monclient: init 2023-10-30T12:10:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7f2c1ca50640 10 monclient: shutdown 2023-10-30T12:10:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7f2c1ca50640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7f2c1ca50640 1 -- 172.21.15.3:0/3969496969 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2c180ae970 msgr2=0x7f2c181518c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7f2c1ca50640 1 --2- 172.21.15.3:0/3969496969 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2c180ae970 0x7f2c181518c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009e80 tx=0x7f2c1002beb0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7f777659c640 1 --2- 172.21.15.3:0/2982470888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77780a8e40 0x7f77780ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7f2c1ca50640 1 -- 172.21.15.3:0/3969496969 shutdown_connections 2023-10-30T12:10:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7f2c1ca50640 1 --2- 172.21.15.3:0/3969496969 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2c180ae970 0x7f2c181518c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7f2c1ca50640 1 -- 172.21.15.3:0/3969496969 >> 172.21.15.3:0/3969496969 conn(0x7f2c180afd30 msgr2=0x7f2c18069d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7f2c1ca50640 1 -- 172.21.15.3:0/3969496969 shutdown_connections 2023-10-30T12:10:20.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7fe2bdd65640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7fe2bdd65640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:20.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7f2c1ca50640 1 -- 172.21.15.3:0/3969496969 wait complete. 2023-10-30T12:10:20.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7fe2bdd65640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2b80a8e40 0x7fe2b81da770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.452+0000 7fe2bdd65640 10 monclient(hunting): picked mon.noname-b con 0x7fe2b80a8e40 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:20.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7fe2bdd65640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2b81423f0 0x7fe2b81dacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7fe2bdd65640 10 monclient(hunting): picked mon.noname-a con 0x7fe2b81423f0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:20.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7fe2bdd65640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2b8142d60 0x7fe2b81431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7fe2bdd65640 10 monclient(hunting): picked mon.noname-c con 0x7fe2b8142d60 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7fe2bdd65640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7fe2bdd65640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b80a83d0 con 0x7fe2b81423f0 2023-10-30T12:10:20.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7fe2bdd65640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7fe2bdd65640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b8144ab0 con 0x7fe2b80a8e40 2023-10-30T12:10:20.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7fe2bdd65640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7fe2bdd65640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b81da250 con 0x7fe2b8142d60 2023-10-30T12:10:20.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7fe2bdd65640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7fe2bdd65640 10 monclient(hunting): authenticate will time out at 1411.667341s 2023-10-30T12:10:20.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f777659c640 10 monclient(hunting): get_auth_request con 0x7f77780a8e40 auth_method 0 2023-10-30T12:10:20.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f777659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f777659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f777659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06a559a640 1 -- 172.21.15.3:0/981438618 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0033050 con 0x7f06a81521d0 2023-10-30T12:10:20.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06a559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:20.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06a559a640 10 monclient: dump: 2023-10-30T12:10:20.651 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.651 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.652 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.652 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.652 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.652 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.652 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.653 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.653 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.653 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06a559a640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06a559a640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06a559a640 1 -- 172.21.15.3:0/981438618 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a000a2d0 con 0x7f06a81521d0 2023-10-30T12:10:20.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06ac9c2640 5 monclient: authenticate success, global_id 4229 2023-10-30T12:10:20.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06ac9c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:20.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06a559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06a559a640 1 -- 172.21.15.3:0/981438618 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a000a670 con 0x7f06a81521d0 2023-10-30T12:10:20.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06ac9c2640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:20.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06ac9c2640 4 set_mon_vals no callback set 2023-10-30T12:10:20.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06a559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:20.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f06a559a640 10 monclient: dump: 2023-10-30T12:10:20.656 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.656 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.656 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.656 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.657 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.657 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.657 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.657 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.657 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.657 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f2c1ca50640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.453+0000 7f2c1ca50640 10 monclient: monmap: 2023-10-30T12:10:20.659 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.659 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:20.659 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:20.659 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:20.659 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.660 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:20.660 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:20.660 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:20.660 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:20.660 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f2c1ca50640 1 librados: starting msgr at 2023-10-30T12:10:20.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f2c1ca50640 1 librados: starting objecter 2023-10-30T12:10:20.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f7775d9b640 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2023-10-30T12:10:20.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f7775d9b640 1 -- 172.21.15.3:0/2982470888 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f77780ab170 msgr2=0x7f77781db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f7775d9b640 1 --2- 172.21.15.3:0/2982470888 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f77780ab170 0x7f77781db250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f7775d9b640 1 -- 172.21.15.3:0/2982470888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77780a8e40 msgr2=0x7f77780ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f7775d9b640 1 --2- 172.21.15.3:0/2982470888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77780a8e40 0x7f77780ae550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f7775d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f7775d9b640 1 monclient: found mon.noname-a 2023-10-30T12:10:20.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f7775d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f7775d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:20.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f7775d9b640 1 -- 172.21.15.3:0/2982470888 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77780ac610 con 0x7f77780abba0 2023-10-30T12:10:20.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f2c1ca50640 1 Processor -- start 2023-10-30T12:10:20.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f06ac9c2640 10 monclient: shutdown 2023-10-30T12:10:20.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f06ac9c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f06ac9c2640 1 -- 172.21.15.3:0/981438618 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f06a81521d0 msgr2=0x7f06a806ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f06ac9c2640 1 --2- 172.21.15.3:0/981438618 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f06a81521d0 0x7f06a806ecf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f06a000c2e0 tx=0x7f06a0030570 comp rx=0 tx=0).stop 2023-10-30T12:10:20.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f2c1ca50640 1 -- start start 2023-10-30T12:10:20.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f2c1ca50640 1 librados: setting wanted keys 2023-10-30T12:10:20.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f2c1ca50640 1 librados: calling monclient init 2023-10-30T12:10:20.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f2c1ca50640 10 monclient: init 2023-10-30T12:10:20.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f06ac9c2640 1 -- 172.21.15.3:0/981438618 shutdown_connections 2023-10-30T12:10:20.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f06ac9c2640 1 --2- 172.21.15.3:0/981438618 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f06a81521d0 0x7f06a806ecf0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f06ac9c2640 1 --2- 172.21.15.3:0/981438618 >> v2:172.21.15.3:3300/0 conn(0x7f06a8151580 0x7f06a80ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f06ac9c2640 1 -- 172.21.15.3:0/981438618 >> 172.21.15.3:0/981438618 conn(0x7f06a80afba0 msgr2=0x7f06a80affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f06ac9c2640 1 -- 172.21.15.3:0/981438618 shutdown_connections 2023-10-30T12:10:20.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.454+0000 7f06ac9c2640 1 -- 172.21.15.3:0/981438618 wait complete. 2023-10-30T12:10:20.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:10:20.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2c180ab220 0x7f2c18148440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 10 monclient(hunting): picked mon.noname-c con 0x7f2c180ab220 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2c180ae530 0x7f2c180a8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 10 monclient(hunting): picked mon.noname-b con 0x7f2c180ae530 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:20.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f36927fc640 1 -- 172.21.15.3:0/3650660865 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368c02cc30 con 0x7f3694143930 2023-10-30T12:10:20.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f36927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f36927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:20.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f36927fc640 10 monclient: dump: 2023-10-30T12:10:20.671 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.671 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.671 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.671 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.672 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.672 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.672 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.672 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.672 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.673 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f36927fc640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f36927fc640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f36927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f36927fc640 1 -- 172.21.15.3:0/3650660865 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368c02cda0 con 0x7f3694143930 2023-10-30T12:10:20.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f36927fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f36927fc640 1 -- 172.21.15.3:0/3650660865 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368c034660 con 0x7f3694143930 2023-10-30T12:10:20.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f36927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f36927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:20.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f36927fc640 10 monclient: dump: 2023-10-30T12:10:20.675 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.675 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.675 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.675 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.675 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.676 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.676 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.676 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.676 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.676 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f3699ffb640 5 monclient: authenticate success, global_id 4238 2023-10-30T12:10:20.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f3699ffb640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:20.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f3699ffb640 4 set_mon_vals no callback set 2023-10-30T12:10:20.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c180ae970 0x7f2c181de3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 10 monclient(hunting): picked mon.noname-a con 0x7f2c180ae970 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:20.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18071720 con 0x7f2c180ae970 2023-10-30T12:10:20.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c180a81e0 con 0x7f2c180ae530 2023-10-30T12:10:20.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c180a88a0 con 0x7f2c180ab220 2023-10-30T12:10:20.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f2c1ca50640 10 monclient(hunting): authenticate will time out at 1411.669505s 2023-10-30T12:10:20.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f06ac9c2640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7f06ac9c2640 10 monclient: monmap: 2023-10-30T12:10:20.681 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.681 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:20.681 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:20.681 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:20.681 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.682 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:20.682 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:20.682 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:20.682 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:20.683 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.455+0000 7fecab7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecac0aa0c0 0x7fecac1ddee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f06ac9c2640 1 librados: starting msgr at 2023-10-30T12:10:20.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f06ac9c2640 1 librados: starting objecter 2023-10-30T12:10:20.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f3699ffb640 10 monclient: shutdown 2023-10-30T12:10:20.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f3699ffb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f3699ffb640 1 -- 172.21.15.3:0/3650660865 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3694143930 msgr2=0x7f36940a8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f3699ffb640 1 --2- 172.21.15.3:0/3650660865 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3694143930 0x7f36940a8cb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f368c009dc0 tx=0x7f368c02be80 comp rx=0 tx=0).stop 2023-10-30T12:10:20.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f3699ffb640 1 -- 172.21.15.3:0/3650660865 shutdown_connections 2023-10-30T12:10:20.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f3699ffb640 1 --2- 172.21.15.3:0/3650660865 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3694143930 0x7f36940a8cb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f3699ffb640 1 -- 172.21.15.3:0/3650660865 >> 172.21.15.3:0/3650660865 conn(0x7f36940b7180 msgr2=0x7f36940b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f06ac9c2640 1 Processor -- start 2023-10-30T12:10:20.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f06ac9c2640 1 -- start start 2023-10-30T12:10:20.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f06ac9c2640 1 librados: setting wanted keys 2023-10-30T12:10:20.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f06ac9c2640 1 librados: calling monclient init 2023-10-30T12:10:20.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f06ac9c2640 10 monclient: init 2023-10-30T12:10:20.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7fecab7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecac0aa0c0 0x7fecac1ddee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56264/0 (socket says 172.21.15.3:56264) 2023-10-30T12:10:20.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7fecab7fe640 1 -- 172.21.15.3:0/1754000992 learned_addr learned my addr 172.21.15.3:0/1754000992 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:20.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7fecab7fe640 10 monclient(hunting): get_auth_request con 0x7fecac0aa0c0 auth_method 0 2023-10-30T12:10:20.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7fecab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7fecab7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7fecab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f3699ffb640 1 -- 172.21.15.3:0/3650660865 shutdown_connections 2023-10-30T12:10:20.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7f3699ffb640 1 -- 172.21.15.3:0/3650660865 wait complete. 2023-10-30T12:10:20.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.456+0000 7fe472746640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46c149810 0x7fe46c14d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f6014ceb640 1 --2- 172.21.15.3:0/3809707824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60101423f0 0x7f601014c320 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f60000097a0 tx=0x7f600002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28b401bee0edeb43 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:20.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06a8151580 0x7f06a807abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 10 monclient(hunting): picked mon.noname-b con 0x7f06a8151580 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:20.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06a8151df0 0x7f06a807b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 10 monclient(hunting): picked mon.noname-a con 0x7f06a8151df0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:20.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f06a81521d0 0x7f06a81e3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 10 monclient(hunting): picked mon.noname-c con 0x7f06a81521d0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f3699ffb640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f3699ffb640 10 monclient: monmap: 2023-10-30T12:10:20.693 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.694 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:20.694 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:20.694 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:20.694 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.695 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:20.695 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:20.695 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:20.695 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:20.695 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a806d1d0 con 0x7f06a8151df0 2023-10-30T12:10:20.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a807bff0 con 0x7f06a8151580 2023-10-30T12:10:20.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a807ccf0 con 0x7f06a81521d0 2023-10-30T12:10:20.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f06ac9c2640 10 monclient(hunting): authenticate will time out at 1411.671361s 2023-10-30T12:10:20.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f3699ffb640 1 librados: starting msgr at 2023-10-30T12:10:20.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f3699ffb640 1 librados: starting objecter 2023-10-30T12:10:20.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.457+0000 7f3699ffb640 1 Processor -- start 2023-10-30T12:10:20.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7fe472746640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46c149810 0x7fe46c14d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56268/0 (socket says 172.21.15.3:56268) 2023-10-30T12:10:20.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7fe472746640 1 -- 172.21.15.3:0/1440719771 learned_addr learned my addr 172.21.15.3:0/1440719771 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:20.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7fe472746640 10 monclient(hunting): get_auth_request con 0x7fe46c149810 auth_method 0 2023-10-30T12:10:20.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7fe472746640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7fe472746640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7fe472746640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 1 -- start start 2023-10-30T12:10:20.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 1 librados: setting wanted keys 2023-10-30T12:10:20.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 1 librados: calling monclient init 2023-10-30T12:10:20.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 10 monclient: init 2023-10-30T12:10:20.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:10:20.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f36940a88d0 0x7f369413eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f36940a88d0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36940b1b00 0x7f36940b6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f36940b1b00 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:20.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3694143930 0x7f36940b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f3694143930 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:20.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36940a8ec0 con 0x7f3694143930 2023-10-30T12:10:20.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36941de1c0 con 0x7f36940b1b00 2023-10-30T12:10:20.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3694145420 con 0x7f36940a88d0 2023-10-30T12:10:20.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7f3699ffb640 10 monclient(hunting): authenticate will time out at 1411.672931s 2023-10-30T12:10:20.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.458+0000 7fecabfff640 1 --2- 172.21.15.3:0/1754000992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecac0ae6b0 0x7fecac0a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7fe2b7fff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2b8142d60 0x7fe2b81431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7fecaaffd640 1 --2- 172.21.15.3:0/1754000992 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecac0ab1d0 0x7fecac0a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7f7775d9b640 1 --2- 172.21.15.3:0/2982470888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77780abba0 0x7f77781dad10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f776c00c440 tx=0x7f776c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7f7766ffd640 10 client.?.objecter ms_handle_connect 0x7f77780abba0 2023-10-30T12:10:20.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7f7766ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:20.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7fecab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7fecab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7fecab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7fecabfff640 10 monclient(hunting): get_auth_request con 0x7fecac0ae6b0 auth_method 0 2023-10-30T12:10:20.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7fecabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7fecabfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7fecabfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7fecaaffd640 10 monclient(hunting): get_auth_request con 0x7fecac0ab1d0 auth_method 0 2023-10-30T12:10:20.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7fecaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7fecaaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.459+0000 7fecaaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.460+0000 7f2c1659c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2c180ae530 0x7f2c180a8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.460+0000 7f2c1659c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2c180ae530 0x7f2c180a8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56298/0 (socket says 172.21.15.3:56298) 2023-10-30T12:10:20.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.460+0000 7f2c1659c640 1 -- 172.21.15.3:0/2032887181 learned_addr learned my addr 172.21.15.3:0/2032887181 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:20.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.460+0000 7f2c1659c640 10 monclient(hunting): get_auth_request con 0x7f2c180ae530 auth_method 0 2023-10-30T12:10:20.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.460+0000 7f2c1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.460+0000 7f2c1659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.460+0000 7f2c1659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.461+0000 7fecaaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.461+0000 7fecaaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.461+0000 7fecaaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.462+0000 7fe2b7fff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2b8142d60 0x7fe2b81431f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:60796/0 (socket says 172.21.15.3:60796) 2023-10-30T12:10:20.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.462+0000 7fe2b7fff640 1 -- 172.21.15.3:0/2306484969 learned_addr learned my addr 172.21.15.3:0/2306484969 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:20.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.462+0000 7fe2b7fff640 10 monclient(hunting): get_auth_request con 0x7fe2b8142d60 auth_method 0 2023-10-30T12:10:20.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.462+0000 7fe2b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.462+0000 7fe2b7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.462+0000 7fe2b7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.463+0000 7fe2b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.463+0000 7fe2b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.463+0000 7fe2b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.463+0000 7fecab7fe640 10 monclient(hunting): handle_auth_done global_id 4300 payload 824 2023-10-30T12:10:20.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.463+0000 7fecab7fe640 1 -- 172.21.15.3:0/1754000992 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecac0ab1d0 msgr2=0x7fecac0a63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.463+0000 7fecab7fe640 1 --2- 172.21.15.3:0/1754000992 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecac0ab1d0 0x7fecac0a63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.463+0000 7fecab7fe640 1 -- 172.21.15.3:0/1754000992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecac0ae6b0 msgr2=0x7fecac0a2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.463+0000 7fecab7fe640 1 --2- 172.21.15.3:0/1754000992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecac0ae6b0 0x7fecac0a2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.463+0000 7fecab7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.463+0000 7fecab7fe640 1 monclient: found mon.noname-b 2023-10-30T12:10:20.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.463+0000 7fecab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.463+0000 7fecab7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.463+0000 7fecab7fe640 1 -- 172.21.15.3:0/1754000992 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac0a8e40 con 0x7fecac0aa0c0 2023-10-30T12:10:20.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7fe472746640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7fe472746640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7fe472746640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f2c1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f2c1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f2c1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f7766ffd640 1 -- 172.21.15.3:0/2982470888 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c0035f0 con 0x7f77780abba0 2023-10-30T12:10:20.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f7766ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f7766ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:20.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f7766ffd640 10 monclient: dump: 2023-10-30T12:10:20.720 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.721 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.721 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.721 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.721 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.721 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.722 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.722 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.722 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.722 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f7766ffd640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f7766ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f7766ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f7766ffd640 1 -- 172.21.15.3:0/2982470888 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776c00e040 con 0x7f77780abba0 2023-10-30T12:10:20.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f7766ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f7766ffd640 1 -- 172.21.15.3:0/2982470888 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c0039c0 con 0x7f77780abba0 2023-10-30T12:10:20.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f7766ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f7766ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:20.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f7766ffd640 10 monclient: dump: 2023-10-30T12:10:20.724 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.724 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.725 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.725 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.725 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.725 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.725 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.725 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.726 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.726 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f777ce2d640 5 monclient: authenticate success, global_id 4332 2023-10-30T12:10:20.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f777ce2d640 10 monclient: _renew_subs 2023-10-30T12:10:20.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f777ce2d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:20.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f777ce2d640 1 -- 172.21.15.3:0/2982470888 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77780b4f80 con 0x7f77780abba0 2023-10-30T12:10:20.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f7774d99640 4 set_mon_vals no callback set 2023-10-30T12:10:20.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f777ce2d640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f777ce2d640 10 monclient: _renew_subs 2023-10-30T12:10:20.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f777ce2d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:20.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f777ce2d640 1 -- 172.21.15.3:0/2982470888 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77780b5450 con 0x7f77780abba0 2023-10-30T12:10:20.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.465+0000 7f777ce2d640 1 librados: init done 2023-10-30T12:10:20.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.466+0000 7f777ce2d640 10 client.4332.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:20.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.466+0000 7f777ce2d640 15 client.4332.objecter _session_command_op_assign -1 1 2023-10-30T12:10:20.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.466+0000 7f777ce2d640 20 client.4332.objecter _get_session osd=-1 returning homeless 2023-10-30T12:10:20.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.466+0000 7f777ce2d640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.466+0000 7f777ce2d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:20.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.466+0000 7f777ce2d640 1 -- 172.21.15.3:0/2982470888 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f77780ac870 con 0x7f77780abba0 2023-10-30T12:10:20.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.466+0000 7fe2b77fe640 1 --2- 172.21.15.3:0/2306484969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2b80a8e40 0x7fe2b81da770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.468+0000 7fe471744640 1 --2- 172.21.15.3:0/1440719771 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe46c0ae7a0 0x7fe46c14cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.469+0000 7fe472746640 10 monclient(hunting): handle_auth_done global_id 4303 payload 824 2023-10-30T12:10:20.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.469+0000 7fe472746640 1 -- 172.21.15.3:0/1440719771 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe46c0ae7a0 msgr2=0x7fe46c14cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.469+0000 7fe472746640 1 --2- 172.21.15.3:0/1440719771 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe46c0ae7a0 0x7fe46c14cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.469+0000 7fe472746640 1 -- 172.21.15.3:0/1440719771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe46c0a9c70 msgr2=0x7fe46c14c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.469+0000 7fe472746640 1 --2- 172.21.15.3:0/1440719771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe46c0a9c70 0x7fe46c14c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.469+0000 7fe472746640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.469+0000 7fe472746640 1 monclient: found mon.noname-b 2023-10-30T12:10:20.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.469+0000 7fe472746640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.469+0000 7fe472746640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.469+0000 7fe472746640 1 -- 172.21.15.3:0/1440719771 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe46c14b320 con 0x7fe46c149810 2023-10-30T12:10:20.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.469+0000 7fe2b77fe640 10 monclient(hunting): get_auth_request con 0x7fe2b80a8e40 auth_method 0 2023-10-30T12:10:20.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.469+0000 7fe2b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.469+0000 7fe2b77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.469+0000 7fe2b77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.472+0000 7f2c1659c640 10 monclient(hunting): handle_auth_done global_id 4306 payload 824 2023-10-30T12:10:20.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.472+0000 7f2c1659c640 1 -- 172.21.15.3:0/2032887181 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2c180ab220 msgr2=0x7f2c18148440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.472+0000 7f2c1659c640 1 --2- 172.21.15.3:0/2032887181 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2c180ab220 0x7f2c18148440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.472+0000 7f2c1659c640 1 -- 172.21.15.3:0/2032887181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c180ae970 msgr2=0x7f2c181de3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.472+0000 7f2c1659c640 1 --2- 172.21.15.3:0/2032887181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c180ae970 0x7f2c181de3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.472+0000 7f2c1659c640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.472+0000 7f2c1659c640 1 monclient: found mon.noname-b 2023-10-30T12:10:20.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.472+0000 7f2c1659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.472+0000 7f2c1659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.472+0000 7f2c1659c640 1 -- 172.21.15.3:0/2032887181 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c180a90e0 con 0x7f2c180ae530 2023-10-30T12:10:20.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.472+0000 7fe2b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.472+0000 7fe2b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.472+0000 7fe2b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.473+0000 7fe2b7fff640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2023-10-30T12:10:20.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.473+0000 7fe2b7fff640 1 -- 172.21.15.3:0/2306484969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2b80a8e40 msgr2=0x7fe2b81da770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.473+0000 7fe2b7fff640 1 --2- 172.21.15.3:0/2306484969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2b80a8e40 0x7fe2b81da770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.473+0000 7fe2b7fff640 1 -- 172.21.15.3:0/2306484969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2b81423f0 msgr2=0x7fe2b81dacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.473+0000 7fe2b7fff640 1 --2- 172.21.15.3:0/2306484969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2b81423f0 0x7fe2b81dacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.473+0000 7fe2b7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.473+0000 7fe2b7fff640 1 monclient: found mon.noname-c 2023-10-30T12:10:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.473+0000 7fe2b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.473+0000 7fe2b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.473+0000 7fe2b7fff640 1 -- 172.21.15.3:0/2306484969 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b80ac9d0 con 0x7fe2b8142d60 2023-10-30T12:10:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.476+0000 7f06a6d9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f06a81521d0 0x7f06a81e3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.476+0000 7fe472746640 1 --2- 172.21.15.3:0/1440719771 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46c149810 0x7fe46c14d280 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe46800c3c0 tx=0x7fe46800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.476+0000 7fe4627fc640 10 client.?.objecter ms_handle_connect 0x7fe46c149810 2023-10-30T12:10:20.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7fe4627fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:20.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7fe2b7fff640 1 --2- 172.21.15.3:0/2306484969 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2b8142d60 0x7fe2b81431f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000cc60 tx=0x7fe2b000a5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7fe29ffff640 10 client.?.objecter ms_handle_connect 0x7fe2b8142d60 2023-10-30T12:10:20.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7fe29ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:20.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f06a6d9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f06a81521d0 0x7f06a81e3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:60814/0 (socket says 172.21.15.3:60814) 2023-10-30T12:10:20.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f06a6d9d640 1 -- 172.21.15.3:0/3510934348 learned_addr learned my addr 172.21.15.3:0/3510934348 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:20.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f06a6d9d640 10 monclient(hunting): get_auth_request con 0x7f06a81521d0 auth_method 0 2023-10-30T12:10:20.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f06a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f06a6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f06a6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f600effd640 1 -- 172.21.15.3:0/3809707824 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6000031700 con 0x7f60101423f0 2023-10-30T12:10:20.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f600effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f600effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:20.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f600effd640 10 monclient: dump: 2023-10-30T12:10:20.746 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.746 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.746 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.747 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.747 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.747 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.747 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.747 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.747 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.748 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f600effd640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f600effd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f600effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f600effd640 1 -- 172.21.15.3:0/3809707824 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6000031870 con 0x7f60101423f0 2023-10-30T12:10:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f601674e640 5 monclient: authenticate success, global_id 4294 2023-10-30T12:10:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f601674e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f600effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f601674e640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f601674e640 4 set_mon_vals no callback set 2023-10-30T12:10:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f600effd640 1 -- 172.21.15.3:0/3809707824 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6000031c10 con 0x7f60101423f0 2023-10-30T12:10:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f600effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f600effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.477+0000 7f600effd640 10 monclient: dump: 2023-10-30T12:10:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.752 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.478+0000 7f601674e640 10 monclient: shutdown 2023-10-30T12:10:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.478+0000 7f601674e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.478+0000 7f601674e640 1 -- 172.21.15.3:0/3809707824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60101423f0 msgr2=0x7f601014c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.478+0000 7f601674e640 1 --2- 172.21.15.3:0/3809707824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60101423f0 0x7f601014c320 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f60000097a0 tx=0x7f600002b7a0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.478+0000 7f601674e640 1 -- 172.21.15.3:0/3809707824 shutdown_connections 2023-10-30T12:10:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.478+0000 7f601674e640 1 --2- 172.21.15.3:0/3809707824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60101423f0 0x7f601014c320 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.478+0000 7f601674e640 1 --2- 172.21.15.3:0/3809707824 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60100b13c0 0x7f6010141eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.478+0000 7f601674e640 1 -- 172.21.15.3:0/3809707824 >> 172.21.15.3:0/3809707824 conn(0x7f60100b7180 msgr2=0x7f60100b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.478+0000 7f601674e640 1 -- 172.21.15.3:0/3809707824 shutdown_connections 2023-10-30T12:10:20.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.478+0000 7f601674e640 1 -- 172.21.15.3:0/3809707824 wait complete. 2023-10-30T12:10:20.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.479+0000 7f601674e640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.479+0000 7f601674e640 10 monclient: monmap: 2023-10-30T12:10:20.755 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:20.755 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:20.755 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:20.755 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:20.755 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.756 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:20.756 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:20.756 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:20.756 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:20.756 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.480+0000 7f601674e640 1 librados: starting msgr at 2023-10-30T12:10:20.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.480+0000 7f601674e640 1 librados: starting objecter 2023-10-30T12:10:20.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.480+0000 7f601674e640 1 Processor -- start 2023-10-30T12:10:20.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.480+0000 7f601674e640 1 -- start start 2023-10-30T12:10:20.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.480+0000 7f601674e640 1 librados: setting wanted keys 2023-10-30T12:10:20.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.480+0000 7f601674e640 1 librados: calling monclient init 2023-10-30T12:10:20.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.480+0000 7f601674e640 10 monclient: init 2023-10-30T12:10:20.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.480+0000 7f601674e640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.480+0000 7f601674e640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:10:20.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.480+0000 7f601674e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60100a8e40 0x7f60100af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.481+0000 7f601674e640 10 monclient(hunting): picked mon.noname-c con 0x7f60100a8e40 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:20.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.481+0000 7f601674e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60101423f0 0x7f6010140730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.481+0000 7f601674e640 10 monclient(hunting): picked mon.noname-b con 0x7f60101423f0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:20.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.481+0000 7f601674e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60100aacb0 0x7f60101de330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.481+0000 7f601674e640 10 monclient(hunting): picked mon.noname-a con 0x7f60100aacb0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:20.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.481+0000 7f601674e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.481+0000 7f601674e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60100aa3c0 con 0x7f60100aacb0 2023-10-30T12:10:20.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.481+0000 7f601674e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.481+0000 7f601674e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6010141430 con 0x7f60101423f0 2023-10-30T12:10:20.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.481+0000 7f601674e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.481+0000 7f601674e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60100ab1d0 con 0x7f60100a8e40 2023-10-30T12:10:20.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.481+0000 7f601674e640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.481+0000 7f601674e640 10 monclient(hunting): authenticate will time out at 1411.695398s 2023-10-30T12:10:20.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.482+0000 7f06a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.482+0000 7f06a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.482+0000 7f06a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7f06a6d9d640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2023-10-30T12:10:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7f06a6d9d640 1 -- 172.21.15.3:0/3510934348 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06a8151580 msgr2=0x7f06a807abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7f06a6d9d640 1 --2- 172.21.15.3:0/3510934348 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06a8151580 0x7f06a807abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7f06a6d9d640 1 -- 172.21.15.3:0/3510934348 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06a8151df0 msgr2=0x7f06a807b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7f06a6d9d640 1 --2- 172.21.15.3:0/3510934348 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06a8151df0 0x7f06a807b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7f06a6d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7f06a6d9d640 1 monclient: found mon.noname-c 2023-10-30T12:10:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7f06a6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7f06a6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7f06a6d9d640 1 -- 172.21.15.3:0/3510934348 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a807cfd0 con 0x7f06a81521d0 2023-10-30T12:10:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7fe4627fc640 1 -- 172.21.15.3:0/1440719771 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4680035f0 con 0x7fe46c149810 2023-10-30T12:10:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7fe4627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7fe4627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7fe4627fc640 10 monclient: dump: 2023-10-30T12:10:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.767 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7fe4627fc640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7fe4627fc640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.485+0000 7fe4627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe472f47640 5 monclient: authenticate success, global_id 4303 2023-10-30T12:10:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe472f47640 10 monclient: _renew_subs 2023-10-30T12:10:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe472f47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe472f47640 1 -- 172.21.15.3:0/1440719771 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe46c14bb10 con 0x7fe46c149810 2023-10-30T12:10:20.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7f3692ffd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36940b1b00 0x7f36940b6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe4627fc640 1 -- 172.21.15.3:0/1440719771 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46800e040 con 0x7fe46c149810 2023-10-30T12:10:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe4627fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe4627fc640 1 -- 172.21.15.3:0/1440719771 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4680039a0 con 0x7fe46c149810 2023-10-30T12:10:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe4627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe472f47640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe4627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:20.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe4627fc640 10 monclient: dump: 2023-10-30T12:10:20.771 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.771 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.772 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.772 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.772 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.772 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.772 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.772 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.773 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.773 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe472f47640 10 monclient: _renew_subs 2023-10-30T12:10:20.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe472f47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe463fff640 4 set_mon_vals no callback set 2023-10-30T12:10:20.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe472f47640 1 -- 172.21.15.3:0/1440719771 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe46c151e70 con 0x7fe46c149810 2023-10-30T12:10:20.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe472f47640 1 librados: init done 2023-10-30T12:10:20.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fecab7fe640 1 --2- 172.21.15.3:0/1754000992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecac0aa0c0 0x7fecac1ddee0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00e8a0 tx=0x7fec9c00ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fec93fff640 10 client.?.objecter ms_handle_connect 0x7fecac0aa0c0 2023-10-30T12:10:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fec93fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe472f47640 10 client.4303.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe472f47640 15 client.4303.objecter _session_command_op_assign -1 1 2023-10-30T12:10:20.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe472f47640 20 client.4303.objecter _get_session osd=-1 returning homeless 2023-10-30T12:10:20.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe472f47640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe472f47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.486+0000 7fe472f47640 1 -- 172.21.15.3:0/1440719771 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe46c0ac530 con 0x7fe46c149810 2023-10-30T12:10:20.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.487+0000 7f2c1659c640 1 --2- 172.21.15.3:0/2032887181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2c180ae530 0x7f2c180a8360 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00c440 tx=0x7f2c0c00c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.487+0000 7f2bff7fe640 10 client.?.objecter ms_handle_connect 0x7f2c180ae530 2023-10-30T12:10:20.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.487+0000 7f2bff7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:20.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.487+0000 7f600ffff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60100a8e40 0x7f60100af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.487+0000 7f36937fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f36940a88d0 0x7f369413eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.488+0000 7f36937fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f36940a88d0 0x7f369413eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:60824/0 (socket says 172.21.15.3:60824) 2023-10-30T12:10:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.488+0000 7f36937fe640 1 -- 172.21.15.3:0/3670555450 learned_addr learned my addr 172.21.15.3:0/3670555450 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.488+0000 7f36937fe640 10 monclient(hunting): get_auth_request con 0x7f36940a88d0 auth_method 0 2023-10-30T12:10:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.488+0000 7f36937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.488+0000 7f36937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.488+0000 7f36937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.488+0000 7f600ffff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60100a8e40 0x7f60100af6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:60826/0 (socket says 172.21.15.3:60826) 2023-10-30T12:10:20.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.489+0000 7f600ffff640 1 -- 172.21.15.3:0/3817402083 learned_addr learned my addr 172.21.15.3:0/3817402083 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:20.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.489+0000 7f600ffff640 10 monclient(hunting): get_auth_request con 0x7f60100a8e40 auth_method 0 2023-10-30T12:10:20.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.489+0000 7f600ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.489+0000 7f600ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.489+0000 7f600ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.489+0000 7f3692ffd640 10 monclient(hunting): get_auth_request con 0x7f36940b1b00 auth_method 0 2023-10-30T12:10:20.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.489+0000 7f3692ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.489+0000 7f3692ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.489+0000 7f3692ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.489+0000 7f600ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.489+0000 7f600ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.489+0000 7f600ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.490+0000 7f06a6d9d640 1 --2- 172.21.15.3:0/3510934348 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f06a81521d0 0x7f06a81e3710 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f069800e960 tx=0x7f069800ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.490+0000 7f0696ffd640 10 client.?.objecter ms_handle_connect 0x7f06a81521d0 2023-10-30T12:10:20.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.490+0000 7f0696ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:20.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.490+0000 7f36937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.490+0000 7f36937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.490+0000 7f36937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.493+0000 7f600ffff640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2023-10-30T12:10:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.493+0000 7f600ffff640 1 -- 172.21.15.3:0/3817402083 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60101423f0 msgr2=0x7f6010140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.493+0000 7f600ffff640 1 --2- 172.21.15.3:0/3817402083 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60101423f0 0x7f6010140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.493+0000 7f600ffff640 1 -- 172.21.15.3:0/3817402083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60100aacb0 msgr2=0x7f60101de330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.493+0000 7f600ffff640 1 --2- 172.21.15.3:0/3817402083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60100aacb0 0x7f60101de330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.493+0000 7f600ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.493+0000 7f600ffff640 1 monclient: found mon.noname-c 2023-10-30T12:10:20.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.493+0000 7f600ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.493+0000 7f600ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.493+0000 7f600ffff640 1 -- 172.21.15.3:0/3817402083 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60101de870 con 0x7f60100a8e40 2023-10-30T12:10:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.494+0000 7f7766ffd640 1 -- 172.21.15.3:0/2982470888 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f776c003b30 con 0x7f77780abba0 2023-10-30T12:10:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.495+0000 7f7766ffd640 1 --2- 172.21.15.3:0/2982470888 >> v2:172.21.15.92:6800/1750155735 conn(0x7f775806c7c0 0x7f775806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.495+0000 7f777659c640 1 --2- 172.21.15.3:0/2982470888 >> v2:172.21.15.92:6800/1750155735 conn(0x7f775806c7c0 0x7f775806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.495+0000 7f777659c640 10 monclient: get_auth_request con 0x7f775806c7c0 auth_method 0 2023-10-30T12:10:20.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.496+0000 7f3692ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.496+0000 7f3692ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.496+0000 7f3692ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.496+0000 7f777659c640 1 --2- 172.21.15.3:0/2982470888 >> v2:172.21.15.92:6800/1750155735 conn(0x7f775806c7c0 0x7f775806ec80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f776800b810 tx=0x7f7768008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.496+0000 7f7766ffd640 10 client.4332.objecter ms_handle_connect 0x7f775806c7c0 2023-10-30T12:10:20.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fec93fff640 1 -- 172.21.15.3:0/1754000992 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c00c940 con 0x7fecac0aa0c0 2023-10-30T12:10:20.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fec93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fec93fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:20.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fec93fff640 10 monclient: dump: 2023-10-30T12:10:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.793 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.793 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.793 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.793 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.793 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fec93fff640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fec93fff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fec93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fec93fff640 1 -- 172.21.15.3:0/1754000992 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9c00d040 con 0x7fecac0aa0c0 2023-10-30T12:10:20.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fec93fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fec93fff640 1 -- 172.21.15.3:0/1754000992 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c00ccf0 con 0x7fecac0aa0c0 2023-10-30T12:10:20.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fecb1f16640 5 monclient: authenticate success, global_id 4300 2023-10-30T12:10:20.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fecb1f16640 10 monclient: _renew_subs 2023-10-30T12:10:20.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fecb1f16640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fecb1f16640 1 -- 172.21.15.3:0/1754000992 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecac1de4b0 con 0x7fecac0aa0c0 2023-10-30T12:10:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7feca9ffb640 4 set_mon_vals no callback set 2023-10-30T12:10:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fec93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fecb1f16640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fec93fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fec93fff640 10 monclient: dump: 2023-10-30T12:10:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.797 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.797 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.797 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.797 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.797 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.798 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.798 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.798 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.798 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fecb1f16640 10 monclient: _renew_subs 2023-10-30T12:10:20.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fecb1f16640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fecb1f16640 1 -- 172.21.15.3:0/1754000992 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecac1de9d0 con 0x7fecac0aa0c0 2023-10-30T12:10:20.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7fecb1f16640 1 librados: init done 2023-10-30T12:10:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2bff7fe640 1 -- 172.21.15.3:0/2032887181 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c005480 con 0x7f2c180ae530 2023-10-30T12:10:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2bff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2bff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2bff7fe640 10 monclient: dump: 2023-10-30T12:10:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.803 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.803 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2bff7fe640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2bff7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2bff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2bff7fe640 1 -- 172.21.15.3:0/2032887181 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c00e040 con 0x7f2c180ae530 2023-10-30T12:10:20.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2bff7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2bff7fe640 1 -- 172.21.15.3:0/2032887181 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c005480 con 0x7f2c180ae530 2023-10-30T12:10:20.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2bff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2bff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:20.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2bff7fe640 10 monclient: dump: 2023-10-30T12:10:20.805 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.805 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.805 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.805 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.805 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.806 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.806 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.806 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.806 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.806 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2c1559a640 4 set_mon_vals no callback set 2023-10-30T12:10:20.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2c1ca50640 5 monclient: authenticate success, global_id 4306 2023-10-30T12:10:20.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2c1ca50640 10 monclient: _renew_subs 2023-10-30T12:10:20.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2c1ca50640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2c1ca50640 1 -- 172.21.15.3:0/2032887181 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c18148b00 con 0x7f2c180ae530 2023-10-30T12:10:20.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2c1ca50640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2c1ca50640 10 monclient: _renew_subs 2023-10-30T12:10:20.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2c1ca50640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2c1ca50640 1 -- 172.21.15.3:0/2032887181 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c181dead0 con 0x7f2c180ae530 2023-10-30T12:10:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.497+0000 7f2c1ca50640 1 librados: init done 2023-10-30T12:10:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7fecb1f16640 10 client.4300.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7fecb1f16640 15 client.4300.objecter _session_command_op_assign -1 1 2023-10-30T12:10:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7fecb1f16640 20 client.4300.objecter _get_session osd=-1 returning homeless 2023-10-30T12:10:20.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7fecb1f16640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7fecb1f16640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7fecb1f16640 1 -- 172.21.15.3:0/1754000992 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fecac0ac760 con 0x7fecac0aa0c0 2023-10-30T12:10:20.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7f2c1ca50640 10 client.4306.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:20.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7f2c1ca50640 15 client.4306.objecter _session_command_op_assign -1 1 2023-10-30T12:10:20.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7f2c1ca50640 20 client.4306.objecter _get_session osd=-1 returning homeless 2023-10-30T12:10:20.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7f2c1ca50640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7f2c1ca50640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:20.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7f2c1ca50640 1 -- 172.21.15.3:0/2032887181 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2c180ac7e0 con 0x7f2c180ae530 2023-10-30T12:10:20.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7f7766ffd640 1 -- 172.21.15.3:0/2982470888 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f776c057e20 con 0x7f77780abba0 2023-10-30T12:10:20.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7f7766ffd640 10 client.4332.objecter ms_dispatch 0x7f7778000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7f7766ffd640 3 client.4332.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.498+0000 7f7766ffd640 3 client.4332.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7766ffd640 10 client.4332.objecter checking command 1 2023-10-30T12:10:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7766ffd640 1 --2- 172.21.15.3:0/2982470888 >> v2:172.21.15.92:6809/3275842310 conn(0x7f77580725d0 0x7f77580749f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7766ffd640 20 client.4332.objecter _get_session s=0x7f776c0039c0 osd=6 3 2023-10-30T12:10:20.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7766ffd640 20 client.4332.objecter put_session s=0x7f776c0039c0 osd=6 3 2023-10-30T12:10:20.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7766ffd640 15 client.4332.objecter _session_command_op_remove -1 1 2023-10-30T12:10:20.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7766ffd640 20 client.4332.objecter _get_session s=0x7f776c0039c0 osd=6 3 2023-10-30T12:10:20.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7766ffd640 20 client.4332.objecter get_session s=0x7f776c0039c0 osd=6 3 2023-10-30T12:10:20.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7766ffd640 15 client.4332.objecter _session_command_op_assign 6 1 2023-10-30T12:10:20.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7766ffd640 20 client.4332.objecter put_session s=0x7f776c0039c0 osd=6 4 2023-10-30T12:10:20.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7766ffd640 10 client.4332.objecter _send_command 1 2023-10-30T12:10:20.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7766ffd640 1 -- 172.21.15.3:0/2982470888 --> v2:172.21.15.92:6809/3275842310 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f776c0035f0 con 0x7f77580725d0 2023-10-30T12:10:20.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7766ffd640 20 client.4332.objecter dump_active .. 0 homeless 2023-10-30T12:10:20.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7766ffd640 1 -- 172.21.15.3:0/2982470888 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f776c053a60 con 0x7f77780abba0 2023-10-30T12:10:20.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7766ffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T12:10:20.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7776d9d640 1 --2- 172.21.15.3:0/2982470888 >> v2:172.21.15.92:6809/3275842310 conn(0x7f77580725d0 0x7f77580749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.499+0000 7f7776d9d640 10 monclient: get_auth_request con 0x7f77580725d0 auth_method 0 2023-10-30T12:10:20.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.500+0000 7f36937fe640 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2023-10-30T12:10:20.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.500+0000 7f36937fe640 1 -- 172.21.15.3:0/3670555450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36940b1b00 msgr2=0x7f36940b6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.500+0000 7f36937fe640 1 --2- 172.21.15.3:0/3670555450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36940b1b00 0x7f36940b6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.500+0000 7f36937fe640 1 -- 172.21.15.3:0/3670555450 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3694143930 msgr2=0x7f36940b6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.500+0000 7f36937fe640 1 --2- 172.21.15.3:0/3670555450 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3694143930 0x7f36940b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.500+0000 7f36937fe640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.500+0000 7f36937fe640 1 monclient: found mon.noname-c 2023-10-30T12:10:20.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.500+0000 7f36937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.500+0000 7f36937fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.500+0000 7f36937fe640 1 -- 172.21.15.3:0/3670555450 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36940b6eb0 con 0x7f36940a88d0 2023-10-30T12:10:20.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.503+0000 7f7766ffd640 10 client.4332.objecter ms_handle_connect 0x7f77580725d0 2023-10-30T12:10:20.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.503+0000 7f7776d9d640 1 --2- 172.21.15.3:0/2982470888 >> v2:172.21.15.92:6809/3275842310 conn(0x7f77580725d0 0x7f77580749f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.503+0000 7f600ffff640 1 --2- 172.21.15.3:0/3817402083 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60100a8e40 0x7f60100af6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f600800cbf0 tx=0x7f600800a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.503+0000 7f600cff9640 10 client.?.objecter ms_handle_connect 0x7f60100a8e40 2023-10-30T12:10:20.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.503+0000 7f600cff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:20.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.505+0000 7f36937fe640 1 --2- 172.21.15.3:0/3670555450 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f36940a88d0 0x7f369413eaf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f368400c700 tx=0x7f368400cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.505+0000 7f367bfff640 10 client.?.objecter ms_handle_connect 0x7f36940a88d0 2023-10-30T12:10:20.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.505+0000 7f367bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:20.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f0696ffd640 1 -- 172.21.15.3:0/3510934348 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069800c980 con 0x7f06a81521d0 2023-10-30T12:10:20.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f0696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f0696ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:20.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f0696ffd640 10 monclient: dump: 2023-10-30T12:10:20.824 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.824 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.824 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.824 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.824 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.825 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.825 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.825 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.825 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.826 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f0696ffd640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f0696ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f0696ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f0696ffd640 1 -- 172.21.15.3:0/3510934348 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069800d040 con 0x7f06a81521d0 2023-10-30T12:10:20.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f0696ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f0696ffd640 1 -- 172.21.15.3:0/3510934348 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069800cd30 con 0x7f06a81521d0 2023-10-30T12:10:20.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f0696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f0696ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:20.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f0696ffd640 10 monclient: dump: 2023-10-30T12:10:20.828 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.828 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.828 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.828 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.828 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.829 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.829 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.829 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.829 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.829 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f06ac9c2640 5 monclient: authenticate success, global_id 4250 2023-10-30T12:10:20.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f06ac9c2640 10 monclient: _renew_subs 2023-10-30T12:10:20.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f06a4d99640 4 set_mon_vals no callback set 2023-10-30T12:10:20.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f06ac9c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f06ac9c2640 1 -- 172.21.15.3:0/3510934348 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a807b7b0 con 0x7f06a81521d0 2023-10-30T12:10:20.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f06ac9c2640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f06ac9c2640 10 monclient: _renew_subs 2023-10-30T12:10:20.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f06ac9c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f06ac9c2640 1 -- 172.21.15.3:0/3510934348 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a81e3e30 con 0x7f06a81521d0 2023-10-30T12:10:20.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.506+0000 7f06ac9c2640 1 librados: init done 2023-10-30T12:10:20.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.507+0000 7f06ac9c2640 10 client.4250.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:20.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.507+0000 7f06ac9c2640 15 client.4250.objecter _session_command_op_assign -1 1 2023-10-30T12:10:20.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.507+0000 7f06ac9c2640 20 client.4250.objecter _get_session osd=-1 returning homeless 2023-10-30T12:10:20.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.507+0000 7f06ac9c2640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.507+0000 7f06ac9c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.507+0000 7f06ac9c2640 1 -- 172.21.15.3:0/3510934348 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f06a80aebd0 con 0x7f06a81521d0 2023-10-30T12:10:20.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe29ffff640 1 -- 172.21.15.3:0/2306484969 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b000ae80 con 0x7fe2b8142d60 2023-10-30T12:10:20.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe29ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe29ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:20.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe29ffff640 10 monclient: dump: 2023-10-30T12:10:20.835 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.835 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.835 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.835 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.835 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.836 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.836 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.836 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.836 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.836 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe29ffff640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe29ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe29ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe29ffff640 1 -- 172.21.15.3:0/2306484969 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b000ce70 con 0x7fe2b8142d60 2023-10-30T12:10:20.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe2bdd65640 5 monclient: authenticate success, global_id 4247 2023-10-30T12:10:20.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe29ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe2bdd65640 10 monclient: _renew_subs 2023-10-30T12:10:20.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe29ffff640 1 -- 172.21.15.3:0/2306484969 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0003bb0 con 0x7fe2b8142d60 2023-10-30T12:10:20.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe2bdd65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe2bdd65640 1 -- 172.21.15.3:0/2306484969 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b8143730 con 0x7fe2b8142d60 2023-10-30T12:10:20.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe29ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe29ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:20.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe29ffff640 10 monclient: dump: 2023-10-30T12:10:20.839 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.839 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.840 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.840 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.840 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.840 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.840 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.840 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.841 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.841 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe2b5ffb640 4 set_mon_vals no callback set 2023-10-30T12:10:20.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe2bdd65640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe2bdd65640 10 monclient: _renew_subs 2023-10-30T12:10:20.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe2bdd65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe2bdd65640 1 -- 172.21.15.3:0/2306484969 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b81d6de0 con 0x7fe2b8142d60 2023-10-30T12:10:20.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.519+0000 7fe2bdd65640 1 librados: init done 2023-10-30T12:10:20.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.520+0000 7fe2bdd65640 10 client.4247.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:20.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.520+0000 7fe2bdd65640 15 client.4247.objecter _session_command_op_assign -1 1 2023-10-30T12:10:20.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.520+0000 7fe2bdd65640 20 client.4247.objecter _get_session osd=-1 returning homeless 2023-10-30T12:10:20.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.520+0000 7fe2bdd65640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.520+0000 7fe2bdd65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.520+0000 7fe2bdd65640 1 -- 172.21.15.3:0/2306484969 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe2b80a97b0 con 0x7fe2b8142d60 2023-10-30T12:10:20.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.532+0000 7f0696ffd640 1 -- 172.21.15.3:0/3510934348 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f0698013470 con 0x7f06a81521d0 2023-10-30T12:10:20.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.532+0000 7f0696ffd640 1 --2- 172.21.15.3:0/3510934348 >> v2:172.21.15.92:6800/1750155735 conn(0x7f068806c810 0x7f068806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.533+0000 7f06a659c640 1 --2- 172.21.15.3:0/3510934348 >> v2:172.21.15.92:6800/1750155735 conn(0x7f068806c810 0x7f068806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.533+0000 7f06a659c640 10 monclient: get_auth_request con 0x7f068806c810 auth_method 0 2023-10-30T12:10:20.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.533+0000 7f0696ffd640 10 client.4250.objecter ms_handle_connect 0x7f068806c810 2023-10-30T12:10:20.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.533+0000 7f06a659c640 1 --2- 172.21.15.3:0/3510934348 >> v2:172.21.15.92:6800/1750155735 conn(0x7f068806c810 0x7f068806ecd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f069c005cd0 tx=0x7f069c0048d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f600cff9640 1 -- 172.21.15.3:0/3817402083 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600800acf0 con 0x7f60100a8e40 2023-10-30T12:10:20.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f600cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f600cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:20.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f600cff9640 10 monclient: dump: 2023-10-30T12:10:20.847 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.847 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.847 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.847 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.848 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.848 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.848 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.848 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.848 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.849 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f600cff9640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f600cff9640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f600cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f600cff9640 1 -- 172.21.15.3:0/3817402083 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600800e040 con 0x7f60100a8e40 2023-10-30T12:10:20.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f601674e640 5 monclient: authenticate success, global_id 4253 2023-10-30T12:10:20.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f601674e640 10 monclient: _renew_subs 2023-10-30T12:10:20.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f601674e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f601674e640 1 -- 172.21.15.3:0/3817402083 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60101deb30 con 0x7f60100a8e40 2023-10-30T12:10:20.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f600cff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f600e7fc640 4 set_mon_vals no callback set 2023-10-30T12:10:20.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f600cff9640 1 -- 172.21.15.3:0/3817402083 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008012640 con 0x7f60100a8e40 2023-10-30T12:10:20.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f600cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f600cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:20.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f601674e640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f600cff9640 10 monclient: dump: 2023-10-30T12:10:20.852 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.852 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.852 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.852 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.852 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.853 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.853 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.853 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.853 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.854 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f601674e640 10 monclient: _renew_subs 2023-10-30T12:10:20.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f601674e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f601674e640 1 -- 172.21.15.3:0/3817402083 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60101dae10 con 0x7f60100a8e40 2023-10-30T12:10:20.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.534+0000 7f601674e640 1 librados: init done 2023-10-30T12:10:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.535+0000 7f601674e640 10 client.4253.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.535+0000 7f601674e640 15 client.4253.objecter _session_command_op_assign -1 1 2023-10-30T12:10:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.535+0000 7f601674e640 20 client.4253.objecter _get_session osd=-1 returning homeless 2023-10-30T12:10:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.535+0000 7f601674e640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.535+0000 7f601674e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.535+0000 7f601674e640 1 -- 172.21.15.3:0/3817402083 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f60100a97b0 con 0x7f60100a8e40 2023-10-30T12:10:20.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.536+0000 7fe4627fc640 1 -- 172.21.15.3:0/1440719771 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fe468003b10 con 0x7fe46c149810 2023-10-30T12:10:20.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.536+0000 7fe4627fc640 1 --2- 172.21.15.3:0/1440719771 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe45006c7c0 0x7fe45006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.536+0000 7fe471f45640 1 --2- 172.21.15.3:0/1440719771 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe45006c7c0 0x7fe45006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.537+0000 7fe471f45640 10 monclient: get_auth_request con 0x7fe45006c7c0 auth_method 0 2023-10-30T12:10:20.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.537+0000 7fe471f45640 1 --2- 172.21.15.3:0/1440719771 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe45006c7c0 0x7fe45006ec80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe464035000 tx=0x7fe46402e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.537+0000 7fe4627fc640 10 client.4303.objecter ms_handle_connect 0x7fe45006c7c0 2023-10-30T12:10:20.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 1 -- 172.21.15.3:0/1440719771 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fe4680580f0 con 0x7fe46c149810 2023-10-30T12:10:20.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 10 client.4303.objecter ms_dispatch 0x7fe46c000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:20.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 3 client.4303.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 3 client.4303.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 10 client.4303.objecter checking command 1 2023-10-30T12:10:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 1 --2- 172.21.15.3:0/1440719771 >> v2:172.21.15.3:6800/1407765917 conn(0x7fe4500725d0 0x7fe4500749f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 20 client.4303.objecter _get_session s=0x7fe4680039a0 osd=2 3 2023-10-30T12:10:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 20 client.4303.objecter put_session s=0x7fe4680039a0 osd=2 3 2023-10-30T12:10:20.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 15 client.4303.objecter _session_command_op_remove -1 1 2023-10-30T12:10:20.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 20 client.4303.objecter _get_session s=0x7fe4680039a0 osd=2 3 2023-10-30T12:10:20.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 20 client.4303.objecter get_session s=0x7fe4680039a0 osd=2 3 2023-10-30T12:10:20.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 15 client.4303.objecter _session_command_op_assign 2 1 2023-10-30T12:10:20.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 20 client.4303.objecter put_session s=0x7fe4680039a0 osd=2 4 2023-10-30T12:10:20.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 10 client.4303.objecter _send_command 1 2023-10-30T12:10:20.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 1 -- 172.21.15.3:0/1440719771 --> v2:172.21.15.3:6800/1407765917 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe4680035f0 con 0x7fe4500725d0 2023-10-30T12:10:20.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.540+0000 7fe4627fc640 20 client.4303.objecter dump_active .. 0 homeless 2023-10-30T12:10:20.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7fe471744640 1 --2- 172.21.15.3:0/1440719771 >> v2:172.21.15.3:6800/1407765917 conn(0x7fe4500725d0 0x7fe4500749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 1 -- 172.21.15.3:0/3510934348 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f0698058980 con 0x7f06a81521d0 2023-10-30T12:10:20.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 10 client.4250.objecter ms_dispatch 0x7f06a8000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:20.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 3 client.4250.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:20.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 3 client.4250.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:20.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 10 client.4250.objecter checking command 1 2023-10-30T12:10:20.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 1 --2- 172.21.15.3:0/3510934348 >> v2:172.21.15.92:6801/4229938171 conn(0x7f0688072620 0x7f0688074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 20 client.4250.objecter _get_session s=0x7f069800cd30 osd=4 3 2023-10-30T12:10:20.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 20 client.4250.objecter put_session s=0x7f069800cd30 osd=4 3 2023-10-30T12:10:20.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 15 client.4250.objecter _session_command_op_remove -1 1 2023-10-30T12:10:20.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 20 client.4250.objecter _get_session s=0x7f069800cd30 osd=4 3 2023-10-30T12:10:20.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 20 client.4250.objecter get_session s=0x7f069800cd30 osd=4 3 2023-10-30T12:10:20.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 15 client.4250.objecter _session_command_op_assign 4 1 2023-10-30T12:10:20.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 20 client.4250.objecter put_session s=0x7f069800cd30 osd=4 4 2023-10-30T12:10:20.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 10 client.4250.objecter _send_command 1 2023-10-30T12:10:20.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 1 -- 172.21.15.3:0/3510934348 --> v2:172.21.15.92:6801/4229938171 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f069800c980 con 0x7f0688072620 2023-10-30T12:10:20.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 20 client.4250.objecter dump_active .. 0 homeless 2023-10-30T12:10:20.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 1 -- 172.21.15.3:0/3510934348 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0698009690 con 0x7f06a81521d0 2023-10-30T12:10:20.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7f0696ffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T12:10:20.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.541+0000 7fe471744640 10 monclient: get_auth_request con 0x7fe4500725d0 auth_method 0 2023-10-30T12:10:20.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.542+0000 7f06a5d9b640 1 --2- 172.21.15.3:0/3510934348 >> v2:172.21.15.92:6801/4229938171 conn(0x7f0688072620 0x7f0688074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.542+0000 7f06a5d9b640 10 monclient: get_auth_request con 0x7f0688072620 auth_method 0 2023-10-30T12:10:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.545+0000 7fe4627fc640 10 client.4303.objecter ms_handle_connect 0x7fe4500725d0 2023-10-30T12:10:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.545+0000 7fe471744640 1 --2- 172.21.15.3:0/1440719771 >> v2:172.21.15.3:6800/1407765917 conn(0x7fe4500725d0 0x7fe4500749f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.546+0000 7f0696ffd640 10 client.4250.objecter ms_handle_connect 0x7f0688072620 2023-10-30T12:10:20.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.546+0000 7f06a5d9b640 1 --2- 172.21.15.3:0/3510934348 >> v2:172.21.15.92:6801/4229938171 conn(0x7f0688072620 0x7f0688074a40 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.551+0000 7f7766ffd640 1 -- 172.21.15.3:0/2982470888 <== osd.6 v2:172.21.15.92:6809/3275842310 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f776c0035f0 con 0x7f77580725d0 2023-10-30T12:10:20.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.551+0000 7f7766ffd640 10 client.4332.objecter ms_dispatch 0x7f7778000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:10:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.551+0000 7f7766ffd640 10 client.4332.objecter _finish_command 1 = system:0 2023-10-30T12:10:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.551+0000 7f7766ffd640 20 client.4332.objecter put_session s=0x7f776c0039c0 osd=6 4 2023-10-30T12:10:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.551+0000 7f7766ffd640 15 client.4332.objecter _session_command_op_remove 6 1 2023-10-30T12:10:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.553+0000 7fe4627fc640 1 -- 172.21.15.3:0/1440719771 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4680010c0 con 0x7fe46c149810 2023-10-30T12:10:20.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.553+0000 7fe4627fc640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T12:10:20.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.554+0000 7fec93fff640 1 -- 172.21.15.3:0/1754000992 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fec9c013470 con 0x7fecac0aa0c0 2023-10-30T12:10:20.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.555+0000 7fec93fff640 1 --2- 172.21.15.3:0/1754000992 >> v2:172.21.15.92:6800/1750155735 conn(0x7fec9806c7c0 0x7fec9806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.555+0000 7fecaaffd640 1 --2- 172.21.15.3:0/1754000992 >> v2:172.21.15.92:6800/1750155735 conn(0x7fec9806c7c0 0x7fec9806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.555+0000 7fecaaffd640 10 monclient: get_auth_request con 0x7fec9806c7c0 auth_method 0 2023-10-30T12:10:20.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.556+0000 7fecaaffd640 1 --2- 172.21.15.3:0/1754000992 >> v2:172.21.15.92:6800/1750155735 conn(0x7fec9806c7c0 0x7fec9806ec80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fecac149bf0 tx=0x7feca400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.556+0000 7fec93fff640 10 client.4300.objecter ms_handle_connect 0x7fec9806c7c0 2023-10-30T12:10:20.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.557+0000 7fec93fff640 1 -- 172.21.15.3:0/1754000992 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fec9c00ce60 con 0x7fecac0aa0c0 2023-10-30T12:10:20.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.557+0000 7fec93fff640 10 client.4300.objecter ms_dispatch 0x7fecac000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:20.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.557+0000 7fec93fff640 3 client.4300.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:20.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.557+0000 7fec93fff640 3 client.4300.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:20.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fec93fff640 10 client.4300.objecter checking command 1 2023-10-30T12:10:20.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fec93fff640 1 --2- 172.21.15.3:0/1754000992 >> v2:172.21.15.3:6812/2654726311 conn(0x7fec980725d0 0x7fec980749f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fec93fff640 20 client.4300.objecter _get_session s=0x7fec9c00ccf0 osd=3 3 2023-10-30T12:10:20.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fec93fff640 20 client.4300.objecter put_session s=0x7fec9c00ccf0 osd=3 3 2023-10-30T12:10:20.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fec93fff640 15 client.4300.objecter _session_command_op_remove -1 1 2023-10-30T12:10:20.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fec93fff640 20 client.4300.objecter _get_session s=0x7fec9c00ccf0 osd=3 3 2023-10-30T12:10:20.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fec93fff640 20 client.4300.objecter get_session s=0x7fec9c00ccf0 osd=3 3 2023-10-30T12:10:20.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fec93fff640 15 client.4300.objecter _session_command_op_assign 3 1 2023-10-30T12:10:20.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fec93fff640 20 client.4300.objecter put_session s=0x7fec9c00ccf0 osd=3 4 2023-10-30T12:10:20.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fec93fff640 10 client.4300.objecter _send_command 1 2023-10-30T12:10:20.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fec93fff640 1 -- 172.21.15.3:0/1754000992 --> v2:172.21.15.3:6812/2654726311 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec9c00c940 con 0x7fec980725d0 2023-10-30T12:10:20.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fec93fff640 20 client.4300.objecter dump_active .. 0 homeless 2023-10-30T12:10:20.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fec93fff640 1 -- 172.21.15.3:0/1754000992 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec9c0094f0 con 0x7fecac0aa0c0 2023-10-30T12:10:20.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fec93fff640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T12:10:20.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fecabfff640 1 --2- 172.21.15.3:0/1754000992 >> v2:172.21.15.3:6812/2654726311 conn(0x7fec980725d0 0x7fec980749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.558+0000 7fecabfff640 10 monclient: get_auth_request con 0x7fec980725d0 auth_method 0 2023-10-30T12:10:20.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f367bfff640 1 -- 172.21.15.3:0/3670555450 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36840052d0 con 0x7f36940a88d0 2023-10-30T12:10:20.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f367bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f367bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:20.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f367bfff640 10 monclient: dump: 2023-10-30T12:10:20.880 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.880 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.880 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.882 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.882 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f367bfff640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f367bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f367bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f367bfff640 1 -- 172.21.15.3:0/3670555450 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368400e040 con 0x7f36940a88d0 2023-10-30T12:10:20.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f367bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f367bfff640 1 -- 172.21.15.3:0/3670555450 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3684003830 con 0x7f36940a88d0 2023-10-30T12:10:20.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f367bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f367bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:20.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f367bfff640 10 monclient: dump: 2023-10-30T12:10:20.884 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:20.884 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:20.884 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.884 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:20.885 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.885 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:20.885 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:20.885 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:20.885 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:20.886 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:20.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f3691ffb640 4 set_mon_vals no callback set 2023-10-30T12:10:20.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f3699ffb640 5 monclient: authenticate success, global_id 4256 2023-10-30T12:10:20.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f3699ffb640 10 monclient: _renew_subs 2023-10-30T12:10:20.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f3699ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f3699ffb640 1 -- 172.21.15.3:0/3670555450 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3694145b20 con 0x7f36940a88d0 2023-10-30T12:10:20.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f3699ffb640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f3699ffb640 10 monclient: _renew_subs 2023-10-30T12:10:20.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f3699ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f3699ffb640 1 -- 172.21.15.3:0/3670555450 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36941442e0 con 0x7f36940a88d0 2023-10-30T12:10:20.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.561+0000 7f3699ffb640 1 librados: init done 2023-10-30T12:10:20.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.562+0000 7f3699ffb640 10 client.4256.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:20.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.562+0000 7f3699ffb640 15 client.4256.objecter _session_command_op_assign -1 1 2023-10-30T12:10:20.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.562+0000 7f3699ffb640 20 client.4256.objecter _get_session osd=-1 returning homeless 2023-10-30T12:10:20.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.562+0000 7f3699ffb640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.562+0000 7f3699ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:20.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.562+0000 7f3699ffb640 1 -- 172.21.15.3:0/3670555450 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f36940aa4c0 con 0x7f36940a88d0 2023-10-30T12:10:20.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.563+0000 7fec93fff640 10 client.4300.objecter ms_handle_connect 0x7fec980725d0 2023-10-30T12:10:20.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.563+0000 7fecabfff640 1 --2- 172.21.15.3:0/1754000992 >> v2:172.21.15.3:6812/2654726311 conn(0x7fec980725d0 0x7fec980749f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.572+0000 7f777ce2d640 10 client.4332.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:10:20.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.572+0000 7f777ce2d640 15 client.4332.objecter _session_command_op_assign -1 2 2023-10-30T12:10:20.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.572+0000 7f777ce2d640 20 client.4332.objecter _get_session s=0x7f776c0039c0 osd=6 3 2023-10-30T12:10:20.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.572+0000 7f777ce2d640 20 client.4332.objecter put_session s=0x7f776c0039c0 osd=6 3 2023-10-30T12:10:20.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.572+0000 7f777ce2d640 20 client.4332.objecter _get_session s=0x7f776c0039c0 osd=6 3 2023-10-30T12:10:20.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.572+0000 7f777ce2d640 15 client.4332.objecter _session_command_op_remove -1 2 2023-10-30T12:10:20.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.572+0000 7f777ce2d640 20 client.4332.objecter get_session s=0x7f776c0039c0 osd=6 3 2023-10-30T12:10:20.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.572+0000 7f777ce2d640 15 client.4332.objecter _session_command_op_assign 6 2 2023-10-30T12:10:20.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.572+0000 7f777ce2d640 20 client.4332.objecter put_session s=0x7f776c0039c0 osd=6 4 2023-10-30T12:10:20.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.572+0000 7f777ce2d640 10 client.4332.objecter _send_command 2 2023-10-30T12:10:20.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.572+0000 7f777ce2d640 1 -- 172.21.15.3:0/2982470888 --> v2:172.21.15.92:6809/3275842310 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f77780ac870 con 0x7f77580725d0 2023-10-30T12:10:20.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.574+0000 7f2bff7fe640 1 -- 172.21.15.3:0/2032887181 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f2c0c0035f0 con 0x7f2c180ae530 2023-10-30T12:10:20.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.574+0000 7f2bff7fe640 1 --2- 172.21.15.3:0/2032887181 >> v2:172.21.15.92:6800/1750155735 conn(0x7f2c0406c880 0x7f2c0406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.574+0000 7f2c16d9d640 1 --2- 172.21.15.3:0/2032887181 >> v2:172.21.15.92:6800/1750155735 conn(0x7f2c0406c880 0x7f2c0406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.574+0000 7f2c16d9d640 10 monclient: get_auth_request con 0x7f2c0406c880 auth_method 0 2023-10-30T12:10:20.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.575+0000 7f2bff7fe640 10 client.4306.objecter ms_handle_connect 0x7f2c0406c880 2023-10-30T12:10:20.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.575+0000 7f2c16d9d640 1 --2- 172.21.15.3:0/2032887181 >> v2:172.21.15.92:6800/1750155735 conn(0x7f2c0406c880 0x7f2c0406ed40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2c08001560 tx=0x7f2c08007990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.576+0000 7f600cff9640 1 -- 172.21.15.3:0/3817402083 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f6008007070 con 0x7f60100a8e40 2023-10-30T12:10:20.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.576+0000 7f600cff9640 1 --2- 172.21.15.3:0/3817402083 >> v2:172.21.15.92:6800/1750155735 conn(0x7f5ff006c810 0x7f5ff006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.577+0000 7f600f7fe640 1 --2- 172.21.15.3:0/3817402083 >> v2:172.21.15.92:6800/1750155735 conn(0x7f5ff006c810 0x7f5ff006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.577+0000 7f600f7fe640 10 monclient: get_auth_request con 0x7f5ff006c810 auth_method 0 2023-10-30T12:10:20.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.577+0000 7f600cff9640 10 client.4253.objecter ms_handle_connect 0x7f5ff006c810 2023-10-30T12:10:20.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.577+0000 7f600f7fe640 1 --2- 172.21.15.3:0/3817402083 >> v2:172.21.15.92:6800/1750155735 conn(0x7f5ff006c810 0x7f5ff006ecd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6010140e10 tx=0x7f5ff8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 1 -- 172.21.15.3:0/2032887181 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f2c0c057620 con 0x7f2c180ae530 2023-10-30T12:10:20.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 10 client.4306.objecter ms_dispatch 0x7f2c18000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:20.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 3 client.4306.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:20.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 3 client.4306.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:20.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 10 client.4306.objecter checking command 1 2023-10-30T12:10:20.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 1 --2- 172.21.15.3:0/2032887181 >> v2:172.21.15.92:6813/1101825516 conn(0x7f2c04072800 0x7f2c04074c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 20 client.4306.objecter _get_session s=0x7f2c04072150 osd=7 3 2023-10-30T12:10:20.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 20 client.4306.objecter put_session s=0x7f2c04072150 osd=7 3 2023-10-30T12:10:20.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 15 client.4306.objecter _session_command_op_remove -1 1 2023-10-30T12:10:20.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 20 client.4306.objecter _get_session s=0x7f2c04072150 osd=7 3 2023-10-30T12:10:20.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 20 client.4306.objecter get_session s=0x7f2c04072150 osd=7 3 2023-10-30T12:10:20.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 15 client.4306.objecter _session_command_op_assign 7 1 2023-10-30T12:10:20.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 20 client.4306.objecter put_session s=0x7f2c04072150 osd=7 4 2023-10-30T12:10:20.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 10 client.4306.objecter _send_command 1 2023-10-30T12:10:20.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 1 -- 172.21.15.3:0/2032887181 --> v2:172.21.15.92:6813/1101825516 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c04075290 con 0x7f2c04072800 2023-10-30T12:10:20.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 20 client.4306.objecter dump_active .. 0 homeless 2023-10-30T12:10:20.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 1 -- 172.21.15.3:0/2032887181 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c180a81e0 con 0x7f2c180ae530 2023-10-30T12:10:20.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2bff7fe640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T12:10:20.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.578+0000 7f2c1759e640 1 --2- 172.21.15.3:0/2032887181 >> v2:172.21.15.92:6813/1101825516 conn(0x7f2c04072800 0x7f2c04074c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.579+0000 7f2c1759e640 10 monclient: get_auth_request con 0x7f2c04072800 auth_method 0 2023-10-30T12:10:20.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f7766ffd640 1 -- 172.21.15.3:0/2982470888 <== osd.6 v2:172.21.15.92:6809/3275842310 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f77780ac870 con 0x7f77580725d0 2023-10-30T12:10:20.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f7766ffd640 10 client.4332.objecter ms_dispatch 0x7f7778000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:10:20.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f7766ffd640 10 client.4332.objecter _finish_command 2 = system:0 2023-10-30T12:10:20.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f7766ffd640 20 client.4332.objecter put_session s=0x7f776c0039c0 osd=6 4 2023-10-30T12:10:20.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f7766ffd640 15 client.4332.objecter _session_command_op_remove 6 2 2023-10-30T12:10:20.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 10 librados: watch_flush enter 2023-10-30T12:10:20.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 10 librados: watch_flush exit 2023-10-30T12:10:20.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 10 client.4332.objecter close_session for osd.6 2023-10-30T12:10:20.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 1 -- 172.21.15.3:0/2982470888 >> v2:172.21.15.92:6809/3275842310 conn(0x7f77580725d0 msgr2=0x7f77580749f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 1 --2- 172.21.15.3:0/2982470888 >> v2:172.21.15.92:6809/3275842310 conn(0x7f77580725d0 0x7f77580749f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 20 client.4332.objecter put_session s=0x7f776c0039c0 osd=6 1 2023-10-30T12:10:20.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 20 client.4332.objecter shutdown clearing up homeless session... 2023-10-30T12:10:20.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 10 client.4332.objecter successfully canceled tick 2023-10-30T12:10:20.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 1 -- 172.21.15.3:0/2982470888 >> v2:172.21.15.92:6800/1750155735 conn(0x7f775806c7c0 msgr2=0x7f775806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 1 --2- 172.21.15.3:0/2982470888 >> v2:172.21.15.92:6800/1750155735 conn(0x7f775806c7c0 0x7f775806ec80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f776800b810 tx=0x7f7768008040 comp rx=0 tx=0).stop 2023-10-30T12:10:20.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 10 monclient: shutdown 2023-10-30T12:10:20.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 1 -- 172.21.15.3:0/2982470888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77780abba0 msgr2=0x7f77781dad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 1 --2- 172.21.15.3:0/2982470888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77780abba0 0x7f77781dad10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f776c00c440 tx=0x7f776c00c900 comp rx=0 tx=0).stop 2023-10-30T12:10:20.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f7776d9d640 1 -- 172.21.15.3:0/2982470888 reap_dead start 2023-10-30T12:10:20.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 1 -- 172.21.15.3:0/2982470888 shutdown_connections 2023-10-30T12:10:20.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 1 -- 172.21.15.3:0/2982470888 >> 172.21.15.3:0/2982470888 conn(0x7f77780b7180 msgr2=0x7f77780b2580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 1 -- 172.21.15.3:0/2982470888 shutdown_connections 2023-10-30T12:10:20.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 1 -- 172.21.15.3:0/2982470888 wait complete. 2023-10-30T12:10:20.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.580+0000 7f777ce2d640 1 librados: shutdown 2023-10-30T12:10:20.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.582+0000 7f2c1759e640 1 --2- 172.21.15.3:0/2032887181 >> v2:172.21.15.92:6813/1101825516 conn(0x7f2c04072800 0x7f2c04074c20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.582+0000 7f2bff7fe640 10 client.4306.objecter ms_handle_connect 0x7f2c04072800 2023-10-30T12:10:20.908 INFO:teuthology.orchestra.run.smithi003.stdout:55834574853 2023-10-30T12:10:20.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T12:10:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.595+0000 7f0696ffd640 1 -- 172.21.15.3:0/3510934348 <== osd.4 v2:172.21.15.92:6801/4229938171 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f069800c980 con 0x7f0688072620 2023-10-30T12:10:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.595+0000 7f0696ffd640 10 client.4250.objecter ms_dispatch 0x7f06a8000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:10:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.595+0000 7f0696ffd640 10 client.4250.objecter _finish_command 1 = system:0 2023-10-30T12:10:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.595+0000 7f0696ffd640 20 client.4250.objecter put_session s=0x7f069800cd30 osd=4 4 2023-10-30T12:10:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.595+0000 7f0696ffd640 15 client.4250.objecter _session_command_op_remove 4 1 2023-10-30T12:10:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.598+0000 7fe29ffff640 1 -- 172.21.15.3:0/2306484969 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fe2b000f3f0 con 0x7fe2b8142d60 2023-10-30T12:10:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.599+0000 7fe29ffff640 1 --2- 172.21.15.3:0/2306484969 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe2a406c810 0x7fe2a406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.599+0000 7fe2b77fe640 1 --2- 172.21.15.3:0/2306484969 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe2a406c810 0x7fe2a406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.599+0000 7fe2b77fe640 10 monclient: get_auth_request con 0x7fe2a406c810 auth_method 0 2023-10-30T12:10:20.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.599+0000 7fe29ffff640 10 client.4247.objecter ms_handle_connect 0x7fe2a406c810 2023-10-30T12:10:20.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.599+0000 7fe2b77fe640 1 --2- 172.21.15.3:0/2306484969 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe2a406c810 0x7fe2a406ecd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe2a80017f0 tx=0x7fe2a8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.602+0000 7fe4627fc640 1 -- 172.21.15.3:0/1440719771 <== osd.2 v2:172.21.15.3:6800/1407765917 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fe4680035f0 con 0x7fe4500725d0 2023-10-30T12:10:20.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.602+0000 7fe4627fc640 10 client.4303.objecter ms_dispatch 0x7fe46c000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:10:20.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.602+0000 7fe4627fc640 10 client.4303.objecter _finish_command 1 = system:0 2023-10-30T12:10:20.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.602+0000 7fe4627fc640 20 client.4303.objecter put_session s=0x7fe4680039a0 osd=2 4 2023-10-30T12:10:20.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.602+0000 7fe4627fc640 15 client.4303.objecter _session_command_op_remove 2 1 2023-10-30T12:10:20.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.604+0000 7fe29ffff640 1 -- 172.21.15.3:0/2306484969 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fe2b0058810 con 0x7fe2b8142d60 2023-10-30T12:10:20.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.604+0000 7fe29ffff640 10 client.4247.objecter ms_dispatch 0x7fe2b8000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:20.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.604+0000 7fe29ffff640 3 client.4247.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:20.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.604+0000 7fe29ffff640 3 client.4247.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:20.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.604+0000 7fe29ffff640 10 client.4247.objecter checking command 1 2023-10-30T12:10:20.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.604+0000 7fe29ffff640 1 --2- 172.21.15.3:0/2306484969 >> v2:172.21.15.3:6804/1445650280 conn(0x7fe2a4072620 0x7fe2a4074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.605+0000 7fe29ffff640 20 client.4247.objecter _get_session s=0x7fe2b0003bb0 osd=1 3 2023-10-30T12:10:20.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.605+0000 7fe29ffff640 20 client.4247.objecter put_session s=0x7fe2b0003bb0 osd=1 3 2023-10-30T12:10:20.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.605+0000 7fe29ffff640 15 client.4247.objecter _session_command_op_remove -1 1 2023-10-30T12:10:20.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.605+0000 7fe29ffff640 20 client.4247.objecter _get_session s=0x7fe2b0003bb0 osd=1 3 2023-10-30T12:10:20.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.605+0000 7fe29ffff640 20 client.4247.objecter get_session s=0x7fe2b0003bb0 osd=1 3 2023-10-30T12:10:20.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.605+0000 7fe29ffff640 15 client.4247.objecter _session_command_op_assign 1 1 2023-10-30T12:10:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.605+0000 7fe29ffff640 20 client.4247.objecter put_session s=0x7fe2b0003bb0 osd=1 4 2023-10-30T12:10:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.605+0000 7fe29ffff640 10 client.4247.objecter _send_command 1 2023-10-30T12:10:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.605+0000 7fe29ffff640 1 -- 172.21.15.3:0/2306484969 --> v2:172.21.15.3:6804/1445650280 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2a40750b0 con 0x7fe2a4072620 2023-10-30T12:10:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.605+0000 7fe29ffff640 20 client.4247.objecter dump_active .. 0 homeless 2023-10-30T12:10:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.605+0000 7fe2b6ffd640 1 --2- 172.21.15.3:0/2306484969 >> v2:172.21.15.3:6804/1445650280 conn(0x7fe2a4072620 0x7fe2a4074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.606+0000 7fe2b6ffd640 10 monclient: get_auth_request con 0x7fe2a4072620 auth_method 0 2023-10-30T12:10:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 1 -- 172.21.15.3:0/3817402083 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f6008058c50 con 0x7f60100a8e40 2023-10-30T12:10:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 10 client.4253.objecter ms_dispatch 0x7f6010000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 3 client.4253.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 3 client.4253.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 10 client.4253.objecter checking command 1 2023-10-30T12:10:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 1 --2- 172.21.15.3:0/3817402083 >> v2:172.21.15.92:6805/3208162957 conn(0x7f5ff0072620 0x7f5ff0074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 20 client.4253.objecter _get_session s=0x7f6008012640 osd=5 3 2023-10-30T12:10:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 20 client.4253.objecter put_session s=0x7f6008012640 osd=5 3 2023-10-30T12:10:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 15 client.4253.objecter _session_command_op_remove -1 1 2023-10-30T12:10:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 20 client.4253.objecter _get_session s=0x7f6008012640 osd=5 3 2023-10-30T12:10:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 20 client.4253.objecter get_session s=0x7f6008012640 osd=5 3 2023-10-30T12:10:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 15 client.4253.objecter _session_command_op_assign 5 1 2023-10-30T12:10:20.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 20 client.4253.objecter put_session s=0x7f6008012640 osd=5 4 2023-10-30T12:10:20.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 10 client.4253.objecter _send_command 1 2023-10-30T12:10:20.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 1 -- 172.21.15.3:0/3817402083 --> v2:172.21.15.92:6805/3208162957 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f600800acf0 con 0x7f5ff0072620 2023-10-30T12:10:20.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f600cff9640 20 client.4253.objecter dump_active .. 0 homeless 2023-10-30T12:10:20.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.608+0000 7f6014ceb640 1 --2- 172.21.15.3:0/3817402083 >> v2:172.21.15.92:6805/3208162957 conn(0x7f5ff0072620 0x7f5ff0074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.609+0000 7f6014ceb640 10 monclient: get_auth_request con 0x7f5ff0072620 auth_method 0 2023-10-30T12:10:20.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.612+0000 7f600cff9640 10 client.4253.objecter ms_handle_connect 0x7f5ff0072620 2023-10-30T12:10:20.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.612+0000 7f6014ceb640 1 --2- 172.21.15.3:0/3817402083 >> v2:172.21.15.92:6805/3208162957 conn(0x7f5ff0072620 0x7f5ff0074a40 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.618+0000 7f06ac9c2640 10 client.4250.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:10:20.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.618+0000 7f06ac9c2640 15 client.4250.objecter _session_command_op_assign -1 2 2023-10-30T12:10:20.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.618+0000 7f06ac9c2640 20 client.4250.objecter _get_session s=0x7f069800cd30 osd=4 3 2023-10-30T12:10:20.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.618+0000 7f06ac9c2640 20 client.4250.objecter put_session s=0x7f069800cd30 osd=4 3 2023-10-30T12:10:20.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.618+0000 7f06ac9c2640 20 client.4250.objecter _get_session s=0x7f069800cd30 osd=4 3 2023-10-30T12:10:20.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.618+0000 7f06ac9c2640 15 client.4250.objecter _session_command_op_remove -1 2 2023-10-30T12:10:20.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.618+0000 7f06ac9c2640 20 client.4250.objecter get_session s=0x7f069800cd30 osd=4 3 2023-10-30T12:10:20.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.618+0000 7f06ac9c2640 15 client.4250.objecter _session_command_op_assign 4 2 2023-10-30T12:10:20.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.618+0000 7f06ac9c2640 20 client.4250.objecter put_session s=0x7f069800cd30 osd=4 4 2023-10-30T12:10:20.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.618+0000 7f06ac9c2640 10 client.4250.objecter _send_command 2 2023-10-30T12:10:20.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.618+0000 7f06ac9c2640 1 -- 172.21.15.3:0/3510934348 --> v2:172.21.15.92:6801/4229938171 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f06a806dfe0 con 0x7f0688072620 2023-10-30T12:10:20.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.619+0000 7fe29ffff640 10 client.4247.objecter ms_handle_connect 0x7fe2a4072620 2023-10-30T12:10:20.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.619+0000 7fe2b6ffd640 1 --2- 172.21.15.3:0/2306484969 >> v2:172.21.15.3:6804/1445650280 conn(0x7fe2a4072620 0x7fe2a4074a40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f0696ffd640 1 -- 172.21.15.3:0/3510934348 <== osd.4 v2:172.21.15.92:6801/4229938171 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f06a000a1d0 con 0x7f0688072620 2023-10-30T12:10:20.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f0696ffd640 10 client.4250.objecter ms_dispatch 0x7f06a8000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:10:20.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f0696ffd640 10 client.4250.objecter _finish_command 2 = system:0 2023-10-30T12:10:20.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f0696ffd640 20 client.4250.objecter put_session s=0x7f069800cd30 osd=4 4 2023-10-30T12:10:20.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f0696ffd640 15 client.4250.objecter _session_command_op_remove 4 2 2023-10-30T12:10:20.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 10 librados: watch_flush enter 2023-10-30T12:10:20.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 10 librados: watch_flush exit 2023-10-30T12:10:20.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 10 client.4250.objecter close_session for osd.4 2023-10-30T12:10:20.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 1 -- 172.21.15.3:0/3510934348 >> v2:172.21.15.92:6801/4229938171 conn(0x7f0688072620 msgr2=0x7f0688074a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 1 --2- 172.21.15.3:0/3510934348 >> v2:172.21.15.92:6801/4229938171 conn(0x7f0688072620 0x7f0688074a40 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 20 client.4250.objecter put_session s=0x7f069800cd30 osd=4 1 2023-10-30T12:10:20.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 20 client.4250.objecter shutdown clearing up homeless session... 2023-10-30T12:10:20.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 10 client.4250.objecter successfully canceled tick 2023-10-30T12:10:20.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 1 -- 172.21.15.3:0/3510934348 >> v2:172.21.15.92:6800/1750155735 conn(0x7f068806c810 msgr2=0x7f068806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 1 --2- 172.21.15.3:0/3510934348 >> v2:172.21.15.92:6800/1750155735 conn(0x7f068806c810 0x7f068806ecd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f069c005cd0 tx=0x7f069c0048d0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 10 monclient: shutdown 2023-10-30T12:10:20.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 1 -- 172.21.15.3:0/3510934348 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f06a81521d0 msgr2=0x7f06a81e3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 1 --2- 172.21.15.3:0/3510934348 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f06a81521d0 0x7f06a81e3710 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f069800e960 tx=0x7f069800ee20 comp rx=0 tx=0).stop 2023-10-30T12:10:20.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06a6d9d640 1 -- 172.21.15.3:0/3510934348 reap_dead start 2023-10-30T12:10:20.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 1 -- 172.21.15.3:0/3510934348 shutdown_connections 2023-10-30T12:10:20.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.625+0000 7f06ac9c2640 1 -- 172.21.15.3:0/3510934348 >> 172.21.15.3:0/3510934348 conn(0x7f06a80afba0 msgr2=0x7f06a8150c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.626+0000 7f06ac9c2640 1 -- 172.21.15.3:0/3510934348 shutdown_connections 2023-10-30T12:10:20.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.626+0000 7f06ac9c2640 1 -- 172.21.15.3:0/3510934348 wait complete. 2023-10-30T12:10:20.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.626+0000 7fec93fff640 1 -- 172.21.15.3:0/1754000992 <== osd.3 v2:172.21.15.3:6812/2654726311 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fec9c00c940 con 0x7fec980725d0 2023-10-30T12:10:20.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.626+0000 7fec93fff640 10 client.4300.objecter ms_dispatch 0x7fecac000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:10:20.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.626+0000 7fec93fff640 10 client.4300.objecter _finish_command 1 = system:0 2023-10-30T12:10:20.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.626+0000 7fec93fff640 20 client.4300.objecter put_session s=0x7fec9c00ccf0 osd=3 4 2023-10-30T12:10:20.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.626+0000 7fec93fff640 15 client.4300.objecter _session_command_op_remove 3 1 2023-10-30T12:10:20.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.626+0000 7f06ac9c2640 1 librados: shutdown 2023-10-30T12:10:20.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.626+0000 7fe29ffff640 1 -- 172.21.15.3:0/2306484969 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2b0054b50 con 0x7fe2b8142d60 2023-10-30T12:10:20.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.626+0000 7fe29ffff640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T12:10:20.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.628+0000 7fe472f47640 10 client.4303.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:10:20.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.628+0000 7fe472f47640 15 client.4303.objecter _session_command_op_assign -1 2 2023-10-30T12:10:20.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.628+0000 7fe472f47640 20 client.4303.objecter _get_session s=0x7fe4680039a0 osd=2 3 2023-10-30T12:10:20.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.628+0000 7fe472f47640 20 client.4303.objecter put_session s=0x7fe4680039a0 osd=2 3 2023-10-30T12:10:20.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.628+0000 7fe472f47640 20 client.4303.objecter _get_session s=0x7fe4680039a0 osd=2 3 2023-10-30T12:10:20.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.628+0000 7fe472f47640 15 client.4303.objecter _session_command_op_remove -1 2 2023-10-30T12:10:20.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.628+0000 7fe472f47640 20 client.4303.objecter get_session s=0x7fe4680039a0 osd=2 3 2023-10-30T12:10:20.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.628+0000 7fe472f47640 15 client.4303.objecter _session_command_op_assign 2 2 2023-10-30T12:10:20.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.628+0000 7fe472f47640 20 client.4303.objecter put_session s=0x7fe4680039a0 osd=2 4 2023-10-30T12:10:20.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.628+0000 7fe472f47640 10 client.4303.objecter _send_command 2 2023-10-30T12:10:20.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.628+0000 7fe472f47640 1 -- 172.21.15.3:0/1440719771 --> v2:172.21.15.3:6800/1407765917 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe46c0ac250 con 0x7fe4500725d0 2023-10-30T12:10:20.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.629+0000 7f600cff9640 1 -- 172.21.15.3:0/3817402083 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6008054890 con 0x7f60100a8e40 2023-10-30T12:10:20.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.629+0000 7f600cff9640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T12:10:20.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.630+0000 7f2bff7fe640 1 -- 172.21.15.3:0/2032887181 <== osd.7 v2:172.21.15.92:6813/1101825516 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f2c04075290 con 0x7f2c04072800 2023-10-30T12:10:20.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.630+0000 7f2bff7fe640 10 client.4306.objecter ms_dispatch 0x7f2c18000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:10:20.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.630+0000 7f2bff7fe640 10 client.4306.objecter _finish_command 1 = system:0 2023-10-30T12:10:20.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.630+0000 7f2bff7fe640 20 client.4306.objecter put_session s=0x7f2c04072150 osd=7 4 2023-10-30T12:10:20.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.630+0000 7f2bff7fe640 15 client.4306.objecter _session_command_op_remove 7 1 2023-10-30T12:10:20.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe4627fc640 1 -- 172.21.15.3:0/1440719771 <== osd.2 v2:172.21.15.3:6800/1407765917 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe45c00a380 con 0x7fe4500725d0 2023-10-30T12:10:20.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe4627fc640 10 client.4303.objecter ms_dispatch 0x7fe46c000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:10:20.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe4627fc640 10 client.4303.objecter _finish_command 2 = system:0 2023-10-30T12:10:20.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe4627fc640 20 client.4303.objecter put_session s=0x7fe4680039a0 osd=2 4 2023-10-30T12:10:20.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe4627fc640 15 client.4303.objecter _session_command_op_remove 2 2 2023-10-30T12:10:20.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe472f47640 10 librados: watch_flush enter 2023-10-30T12:10:20.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe472f47640 10 librados: watch_flush exit 2023-10-30T12:10:20.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe472f47640 10 client.4303.objecter close_session for osd.2 2023-10-30T12:10:20.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe472f47640 1 -- 172.21.15.3:0/1440719771 >> v2:172.21.15.3:6800/1407765917 conn(0x7fe4500725d0 msgr2=0x7fe4500749f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe472f47640 1 --2- 172.21.15.3:0/1440719771 >> v2:172.21.15.3:6800/1407765917 conn(0x7fe4500725d0 0x7fe4500749f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe472f47640 20 client.4303.objecter put_session s=0x7fe4680039a0 osd=2 1 2023-10-30T12:10:20.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe472f47640 20 client.4303.objecter shutdown clearing up homeless session... 2023-10-30T12:10:20.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe472f47640 10 client.4303.objecter successfully canceled tick 2023-10-30T12:10:20.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe472f47640 1 -- 172.21.15.3:0/1440719771 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe45006c7c0 msgr2=0x7fe45006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe472f47640 1 --2- 172.21.15.3:0/1440719771 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe45006c7c0 0x7fe45006ec80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe464035000 tx=0x7fe46402e000 comp rx=0 tx=0).stop 2023-10-30T12:10:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe472f47640 10 monclient: shutdown 2023-10-30T12:10:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.638+0000 7fe472f47640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.639+0000 7fe472f47640 1 -- 172.21.15.3:0/1440719771 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46c149810 msgr2=0x7fe46c14d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.639+0000 7fe472f47640 1 --2- 172.21.15.3:0/1440719771 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46c149810 0x7fe46c14d280 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe46800c3c0 tx=0x7fe46800c880 comp rx=0 tx=0).stop 2023-10-30T12:10:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.639+0000 7fe472746640 1 -- 172.21.15.3:0/1440719771 reap_dead start 2023-10-30T12:10:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.640+0000 7fe472f47640 1 -- 172.21.15.3:0/1440719771 shutdown_connections 2023-10-30T12:10:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.640+0000 7fe472f47640 1 -- 172.21.15.3:0/1440719771 >> 172.21.15.3:0/1440719771 conn(0x7fe46c069d50 msgr2=0x7fe46c06a130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.640+0000 7fe472f47640 1 -- 172.21.15.3:0/1440719771 shutdown_connections 2023-10-30T12:10:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.640+0000 7fe472f47640 1 -- 172.21.15.3:0/1440719771 wait complete. 2023-10-30T12:10:20.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.640+0000 7fe472f47640 1 librados: shutdown 2023-10-30T12:10:20.946 INFO:teuthology.orchestra.run.smithi003.stdout:55834574853 2023-10-30T12:10:20.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T12:10:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.646+0000 7fecb1f16640 10 client.4300.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:10:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.646+0000 7fecb1f16640 15 client.4300.objecter _session_command_op_assign -1 2 2023-10-30T12:10:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.646+0000 7fecb1f16640 20 client.4300.objecter _get_session s=0x7fec9c00ccf0 osd=3 3 2023-10-30T12:10:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.646+0000 7fecb1f16640 20 client.4300.objecter put_session s=0x7fec9c00ccf0 osd=3 3 2023-10-30T12:10:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.646+0000 7fecb1f16640 20 client.4300.objecter _get_session s=0x7fec9c00ccf0 osd=3 3 2023-10-30T12:10:20.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.646+0000 7fecb1f16640 15 client.4300.objecter _session_command_op_remove -1 2 2023-10-30T12:10:20.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.646+0000 7fecb1f16640 20 client.4300.objecter get_session s=0x7fec9c00ccf0 osd=3 3 2023-10-30T12:10:20.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.646+0000 7fecb1f16640 15 client.4300.objecter _session_command_op_assign 3 2 2023-10-30T12:10:20.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.646+0000 7fecb1f16640 20 client.4300.objecter put_session s=0x7fec9c00ccf0 osd=3 4 2023-10-30T12:10:20.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.646+0000 7fecb1f16640 10 client.4300.objecter _send_command 2 2023-10-30T12:10:20.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.646+0000 7fecb1f16640 1 -- 172.21.15.3:0/1754000992 --> v2:172.21.15.3:6812/2654726311 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fecac0ac480 con 0x7fec980725d0 2023-10-30T12:10:20.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.649+0000 7f367bfff640 1 -- 172.21.15.3:0/3670555450 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f36840039a0 con 0x7f36940a88d0 2023-10-30T12:10:20.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.650+0000 7f367bfff640 1 --2- 172.21.15.3:0/3670555450 >> v2:172.21.15.92:6800/1750155735 conn(0x7f368006c810 0x7f368006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.650+0000 7f3692ffd640 1 --2- 172.21.15.3:0/3670555450 >> v2:172.21.15.92:6800/1750155735 conn(0x7f368006c810 0x7f368006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.650+0000 7f3692ffd640 10 monclient: get_auth_request con 0x7f368006c810 auth_method 0 2023-10-30T12:10:20.952 INFO:teuthology.orchestra.run.smithi003.stdout:68719476743 2023-10-30T12:10:20.952 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T12:10:20.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.651+0000 7f3692ffd640 1 --2- 172.21.15.3:0/3670555450 >> v2:172.21.15.92:6800/1750155735 conn(0x7f368006c810 0x7f368006ecd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f36940b3ee0 tx=0x7f3688009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.651+0000 7f367bfff640 10 client.4256.objecter ms_handle_connect 0x7f368006c810 2023-10-30T12:10:20.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fec93fff640 1 -- 172.21.15.3:0/1754000992 <== osd.3 v2:172.21.15.3:6812/2654726311 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7feca0009420 con 0x7fec980725d0 2023-10-30T12:10:20.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fec93fff640 10 client.4300.objecter ms_dispatch 0x7fecac000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:10:20.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fec93fff640 10 client.4300.objecter _finish_command 2 = system:0 2023-10-30T12:10:20.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fec93fff640 20 client.4300.objecter put_session s=0x7fec9c00ccf0 osd=3 4 2023-10-30T12:10:20.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fec93fff640 15 client.4300.objecter _session_command_op_remove 3 2 2023-10-30T12:10:20.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 10 librados: watch_flush enter 2023-10-30T12:10:20.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 10 librados: watch_flush exit 2023-10-30T12:10:20.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 10 client.4300.objecter close_session for osd.3 2023-10-30T12:10:20.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 1 -- 172.21.15.3:0/1754000992 >> v2:172.21.15.3:6812/2654726311 conn(0x7fec980725d0 msgr2=0x7fec980749f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 1 --2- 172.21.15.3:0/1754000992 >> v2:172.21.15.3:6812/2654726311 conn(0x7fec980725d0 0x7fec980749f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 20 client.4300.objecter put_session s=0x7fec9c00ccf0 osd=3 1 2023-10-30T12:10:20.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 20 client.4300.objecter shutdown clearing up homeless session... 2023-10-30T12:10:20.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 10 client.4300.objecter successfully canceled tick 2023-10-30T12:10:20.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 1 -- 172.21.15.3:0/1754000992 >> v2:172.21.15.92:6800/1750155735 conn(0x7fec9806c7c0 msgr2=0x7fec9806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 1 --2- 172.21.15.3:0/1754000992 >> v2:172.21.15.92:6800/1750155735 conn(0x7fec9806c7c0 0x7fec9806ec80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fecac149bf0 tx=0x7feca400b040 comp rx=0 tx=0).stop 2023-10-30T12:10:20.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 10 monclient: shutdown 2023-10-30T12:10:20.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 1 -- 172.21.15.3:0/1754000992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecac0aa0c0 msgr2=0x7fecac1ddee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 1 --2- 172.21.15.3:0/1754000992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecac0aa0c0 0x7fecac1ddee0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00e8a0 tx=0x7fec9c00ed60 comp rx=0 tx=0).stop 2023-10-30T12:10:20.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecabfff640 1 -- 172.21.15.3:0/1754000992 reap_dead start 2023-10-30T12:10:20.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 1 -- 172.21.15.3:0/1754000992 shutdown_connections 2023-10-30T12:10:20.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 1 -- 172.21.15.3:0/1754000992 >> 172.21.15.3:0/1754000992 conn(0x7fecac069d50 msgr2=0x7fecac140cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 1 -- 172.21.15.3:0/1754000992 shutdown_connections 2023-10-30T12:10:20.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 1 -- 172.21.15.3:0/1754000992 wait complete. 2023-10-30T12:10:20.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.656+0000 7fecb1f16640 1 librados: shutdown 2023-10-30T12:10:20.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f600cff9640 1 -- 172.21.15.3:0/3817402083 <== osd.5 v2:172.21.15.92:6805/3208162957 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f600800acf0 con 0x7f5ff0072620 2023-10-30T12:10:20.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f600cff9640 10 client.4253.objecter ms_dispatch 0x7f6010000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:10:20.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f600cff9640 10 client.4253.objecter _finish_command 1 = system:0 2023-10-30T12:10:20.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f600cff9640 20 client.4253.objecter put_session s=0x7f6008012640 osd=5 4 2023-10-30T12:10:20.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f600cff9640 15 client.4253.objecter _session_command_op_remove 5 1 2023-10-30T12:10:20.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 1 -- 172.21.15.3:0/3670555450 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f3684058840 con 0x7f36940a88d0 2023-10-30T12:10:20.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 10 client.4256.objecter ms_dispatch 0x7f3694000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:20.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 3 client.4256.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:20.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 3 client.4256.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:20.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 10 client.4256.objecter checking command 1 2023-10-30T12:10:20.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 1 --2- 172.21.15.3:0/3670555450 >> v2:172.21.15.3:6808/3639283465 conn(0x7f3680072620 0x7f3680074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 20 client.4256.objecter _get_session s=0x7f3684003830 osd=0 3 2023-10-30T12:10:20.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 20 client.4256.objecter put_session s=0x7f3684003830 osd=0 3 2023-10-30T12:10:20.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 15 client.4256.objecter _session_command_op_remove -1 1 2023-10-30T12:10:20.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 20 client.4256.objecter _get_session s=0x7f3684003830 osd=0 3 2023-10-30T12:10:20.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 20 client.4256.objecter get_session s=0x7f3684003830 osd=0 3 2023-10-30T12:10:20.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 15 client.4256.objecter _session_command_op_assign 0 1 2023-10-30T12:10:20.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 20 client.4256.objecter put_session s=0x7f3684003830 osd=0 4 2023-10-30T12:10:20.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 10 client.4256.objecter _send_command 1 2023-10-30T12:10:20.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 1 -- 172.21.15.3:0/3670555450 --> v2:172.21.15.3:6808/3639283465 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36840052d0 con 0x7f3680072620 2023-10-30T12:10:20.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 20 client.4256.objecter dump_active .. 0 homeless 2023-10-30T12:10:20.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 1 -- 172.21.15.3:0/3670555450 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3684001d80 con 0x7f36940a88d0 2023-10-30T12:10:20.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.657+0000 7f367bfff640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T12:10:20.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.658+0000 7f2c1ca50640 10 client.4306.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:10:20.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.658+0000 7f2c1ca50640 15 client.4306.objecter _session_command_op_assign -1 2 2023-10-30T12:10:20.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.658+0000 7f2c1ca50640 20 client.4306.objecter _get_session s=0x7f2c04072150 osd=7 3 2023-10-30T12:10:20.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.658+0000 7f2c1ca50640 20 client.4306.objecter put_session s=0x7f2c04072150 osd=7 3 2023-10-30T12:10:20.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.658+0000 7f2c1ca50640 20 client.4306.objecter _get_session s=0x7f2c04072150 osd=7 3 2023-10-30T12:10:20.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.658+0000 7f2c1ca50640 15 client.4306.objecter _session_command_op_remove -1 2 2023-10-30T12:10:20.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.658+0000 7f2c1ca50640 20 client.4306.objecter get_session s=0x7f2c04072150 osd=7 3 2023-10-30T12:10:20.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.658+0000 7f2c1ca50640 15 client.4306.objecter _session_command_op_assign 7 2 2023-10-30T12:10:20.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.658+0000 7f2c1ca50640 20 client.4306.objecter put_session s=0x7f2c04072150 osd=7 4 2023-10-30T12:10:20.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.658+0000 7f2c1ca50640 10 client.4306.objecter _send_command 2 2023-10-30T12:10:20.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.658+0000 7f2c1ca50640 1 -- 172.21.15.3:0/2032887181 --> v2:172.21.15.92:6813/1101825516 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2c180ac500 con 0x7f2c04072800 2023-10-30T12:10:20.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.658+0000 7f3693fff640 1 --2- 172.21.15.3:0/3670555450 >> v2:172.21.15.3:6808/3639283465 conn(0x7f3680072620 0x7f3680074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.659+0000 7f3693fff640 10 monclient: get_auth_request con 0x7f3680072620 auth_method 0 2023-10-30T12:10:20.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.663+0000 7f367bfff640 10 client.4256.objecter ms_handle_connect 0x7f3680072620 2023-10-30T12:10:20.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.663+0000 7f3693fff640 1 --2- 172.21.15.3:0/3670555450 >> v2:172.21.15.3:6808/3639283465 conn(0x7f3680072620 0x7f3680074a40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.665+0000 7f2bff7fe640 1 -- 172.21.15.3:0/2032887181 <== osd.7 v2:172.21.15.92:6813/1101825516 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2c1002c850 con 0x7f2c04072800 2023-10-30T12:10:20.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.665+0000 7f2bff7fe640 10 client.4306.objecter ms_dispatch 0x7f2c18000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:10:20.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.665+0000 7f2bff7fe640 10 client.4306.objecter _finish_command 2 = system:0 2023-10-30T12:10:20.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.665+0000 7f2bff7fe640 20 client.4306.objecter put_session s=0x7f2c04072150 osd=7 4 2023-10-30T12:10:20.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.665+0000 7f2bff7fe640 15 client.4306.objecter _session_command_op_remove 7 2 2023-10-30T12:10:20.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 10 librados: watch_flush enter 2023-10-30T12:10:20.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 10 librados: watch_flush exit 2023-10-30T12:10:20.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 10 client.4306.objecter close_session for osd.7 2023-10-30T12:10:20.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 1 -- 172.21.15.3:0/2032887181 >> v2:172.21.15.92:6813/1101825516 conn(0x7f2c04072800 msgr2=0x7f2c04074c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 1 --2- 172.21.15.3:0/2032887181 >> v2:172.21.15.92:6813/1101825516 conn(0x7f2c04072800 0x7f2c04074c20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 20 client.4306.objecter put_session s=0x7f2c04072150 osd=7 1 2023-10-30T12:10:20.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 20 client.4306.objecter shutdown clearing up homeless session... 2023-10-30T12:10:20.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 10 client.4306.objecter successfully canceled tick 2023-10-30T12:10:20.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 1 -- 172.21.15.3:0/2032887181 >> v2:172.21.15.92:6800/1750155735 conn(0x7f2c0406c880 msgr2=0x7f2c0406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 1 --2- 172.21.15.3:0/2032887181 >> v2:172.21.15.92:6800/1750155735 conn(0x7f2c0406c880 0x7f2c0406ed40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2c08001560 tx=0x7f2c08007990 comp rx=0 tx=0).stop 2023-10-30T12:10:20.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 10 monclient: shutdown 2023-10-30T12:10:20.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 1 -- 172.21.15.3:0/2032887181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2c180ae530 msgr2=0x7f2c180a8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 1 --2- 172.21.15.3:0/2032887181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2c180ae530 0x7f2c180a8360 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00c440 tx=0x7f2c0c00c900 comp rx=0 tx=0).stop 2023-10-30T12:10:20.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1759e640 1 -- 172.21.15.3:0/2032887181 reap_dead start 2023-10-30T12:10:20.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 1 -- 172.21.15.3:0/2032887181 shutdown_connections 2023-10-30T12:10:20.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 1 -- 172.21.15.3:0/2032887181 >> 172.21.15.3:0/2032887181 conn(0x7f2c180afd30 msgr2=0x7f2c181462f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 1 -- 172.21.15.3:0/2032887181 shutdown_connections 2023-10-30T12:10:20.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 1 -- 172.21.15.3:0/2032887181 wait complete. 2023-10-30T12:10:20.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.666+0000 7f2c1ca50640 1 librados: shutdown 2023-10-30T12:10:20.979 INFO:teuthology.orchestra.run.smithi003.stdout:68719476741 2023-10-30T12:10:20.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T12:10:20.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.674+0000 7f601674e640 10 client.4253.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:10:20.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.674+0000 7f601674e640 15 client.4253.objecter _session_command_op_assign -1 2 2023-10-30T12:10:20.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.674+0000 7f601674e640 20 client.4253.objecter _get_session s=0x7f6008012640 osd=5 3 2023-10-30T12:10:20.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.674+0000 7f601674e640 20 client.4253.objecter put_session s=0x7f6008012640 osd=5 3 2023-10-30T12:10:20.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.674+0000 7f601674e640 20 client.4253.objecter _get_session s=0x7f6008012640 osd=5 3 2023-10-30T12:10:20.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.674+0000 7f601674e640 15 client.4253.objecter _session_command_op_remove -1 2 2023-10-30T12:10:20.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.674+0000 7f601674e640 20 client.4253.objecter get_session s=0x7f6008012640 osd=5 3 2023-10-30T12:10:20.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.674+0000 7f601674e640 15 client.4253.objecter _session_command_op_assign 5 2 2023-10-30T12:10:20.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.674+0000 7f601674e640 20 client.4253.objecter put_session s=0x7f6008012640 osd=5 4 2023-10-30T12:10:20.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.674+0000 7f601674e640 10 client.4253.objecter _send_command 2 2023-10-30T12:10:20.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.674+0000 7f601674e640 1 -- 172.21.15.3:0/3817402083 --> v2:172.21.15.92:6805/3208162957 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f60100a9220 con 0x7f5ff0072620 2023-10-30T12:10:20.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.680+0000 7f600cff9640 1 -- 172.21.15.3:0/3817402083 <== osd.5 v2:172.21.15.92:6805/3208162957 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6000006e10 con 0x7f5ff0072620 2023-10-30T12:10:20.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.680+0000 7f600cff9640 10 client.4253.objecter ms_dispatch 0x7f6010000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:10:20.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.680+0000 7f600cff9640 10 client.4253.objecter _finish_command 2 = system:0 2023-10-30T12:10:20.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.680+0000 7f600cff9640 20 client.4253.objecter put_session s=0x7f6008012640 osd=5 4 2023-10-30T12:10:20.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.680+0000 7f600cff9640 15 client.4253.objecter _session_command_op_remove 5 2 2023-10-30T12:10:20.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.680+0000 7f601674e640 10 librados: watch_flush enter 2023-10-30T12:10:20.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.680+0000 7f601674e640 10 librados: watch_flush exit 2023-10-30T12:10:20.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.680+0000 7f601674e640 10 client.4253.objecter close_session for osd.5 2023-10-30T12:10:20.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.680+0000 7f601674e640 1 -- 172.21.15.3:0/3817402083 >> v2:172.21.15.92:6805/3208162957 conn(0x7f5ff0072620 msgr2=0x7f5ff0074a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.680+0000 7f601674e640 1 --2- 172.21.15.3:0/3817402083 >> v2:172.21.15.92:6805/3208162957 conn(0x7f5ff0072620 0x7f5ff0074a40 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.680+0000 7f601674e640 20 client.4253.objecter put_session s=0x7f6008012640 osd=5 1 2023-10-30T12:10:20.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.680+0000 7f601674e640 20 client.4253.objecter shutdown clearing up homeless session... 2023-10-30T12:10:20.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.680+0000 7f601674e640 10 client.4253.objecter successfully canceled tick 2023-10-30T12:10:20.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.681+0000 7f601674e640 1 -- 172.21.15.3:0/3817402083 >> v2:172.21.15.92:6800/1750155735 conn(0x7f5ff006c810 msgr2=0x7f5ff006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.681+0000 7f601674e640 1 --2- 172.21.15.3:0/3817402083 >> v2:172.21.15.92:6800/1750155735 conn(0x7f5ff006c810 0x7f5ff006ecd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6010140e10 tx=0x7f5ff8008040 comp rx=0 tx=0).stop 2023-10-30T12:10:20.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.681+0000 7f601674e640 10 monclient: shutdown 2023-10-30T12:10:20.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.681+0000 7f601674e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.681+0000 7f601674e640 1 -- 172.21.15.3:0/3817402083 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60100a8e40 msgr2=0x7f60100af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.681+0000 7f601674e640 1 --2- 172.21.15.3:0/3817402083 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60100a8e40 0x7f60100af6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f600800cbf0 tx=0x7f600800a430 comp rx=0 tx=0).stop 2023-10-30T12:10:20.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.681+0000 7f6014ceb640 1 -- 172.21.15.3:0/3817402083 reap_dead start 2023-10-30T12:10:20.989 INFO:teuthology.orchestra.run.smithi003.stdout:60129542150 2023-10-30T12:10:20.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T12:10:20.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.681+0000 7f601674e640 1 -- 172.21.15.3:0/3817402083 shutdown_connections 2023-10-30T12:10:20.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.681+0000 7f601674e640 1 -- 172.21.15.3:0/3817402083 >> 172.21.15.3:0/3817402083 conn(0x7f60100b7180 msgr2=0x7f60100b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.681+0000 7f601674e640 1 -- 172.21.15.3:0/3817402083 shutdown_connections 2023-10-30T12:10:20.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.681+0000 7f601674e640 1 -- 172.21.15.3:0/3817402083 wait complete. 2023-10-30T12:10:20.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.681+0000 7f601674e640 1 librados: shutdown 2023-10-30T12:10:20.992 INFO:teuthology.orchestra.run.smithi003.stdout:55834574853 2023-10-30T12:10:20.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T12:10:20.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.704+0000 7fe29ffff640 1 -- 172.21.15.3:0/2306484969 <== osd.1 v2:172.21.15.3:6804/1445650280 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fe2a40750b0 con 0x7fe2a4072620 2023-10-30T12:10:20.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.704+0000 7fe29ffff640 10 client.4247.objecter ms_dispatch 0x7fe2b8000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:10:20.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.704+0000 7fe29ffff640 10 client.4247.objecter _finish_command 1 = system:0 2023-10-30T12:10:20.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.704+0000 7fe29ffff640 20 client.4247.objecter put_session s=0x7fe2b0003bb0 osd=1 4 2023-10-30T12:10:20.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.704+0000 7fe29ffff640 15 client.4247.objecter _session_command_op_remove 1 1 2023-10-30T12:10:20.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.720+0000 7fe2bdd65640 10 client.4247.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:10:20.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.720+0000 7fe2bdd65640 15 client.4247.objecter _session_command_op_assign -1 2 2023-10-30T12:10:20.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.720+0000 7fe2bdd65640 20 client.4247.objecter _get_session s=0x7fe2b0003bb0 osd=1 3 2023-10-30T12:10:20.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.720+0000 7fe2bdd65640 20 client.4247.objecter put_session s=0x7fe2b0003bb0 osd=1 3 2023-10-30T12:10:20.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.720+0000 7fe2bdd65640 20 client.4247.objecter _get_session s=0x7fe2b0003bb0 osd=1 3 2023-10-30T12:10:20.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.720+0000 7fe2bdd65640 15 client.4247.objecter _session_command_op_remove -1 2 2023-10-30T12:10:20.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.720+0000 7fe2bdd65640 20 client.4247.objecter get_session s=0x7fe2b0003bb0 osd=1 3 2023-10-30T12:10:20.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.720+0000 7fe2bdd65640 15 client.4247.objecter _session_command_op_assign 1 2 2023-10-30T12:10:20.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.720+0000 7fe2bdd65640 20 client.4247.objecter put_session s=0x7fe2b0003bb0 osd=1 4 2023-10-30T12:10:20.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.720+0000 7fe2bdd65640 10 client.4247.objecter _send_command 2 2023-10-30T12:10:20.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.720+0000 7fe2bdd65640 1 -- 172.21.15.3:0/2306484969 --> v2:172.21.15.3:6804/1445650280 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe2b80a9220 con 0x7fe2a4072620 2023-10-30T12:10:20.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe29ffff640 1 -- 172.21.15.3:0/2306484969 <== osd.1 v2:172.21.15.3:6804/1445650280 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe2ac031760 con 0x7fe2a4072620 2023-10-30T12:10:20.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe29ffff640 10 client.4247.objecter ms_dispatch 0x7fe2b8000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:10:20.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe29ffff640 10 client.4247.objecter _finish_command 2 = system:0 2023-10-30T12:10:20.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe29ffff640 20 client.4247.objecter put_session s=0x7fe2b0003bb0 osd=1 4 2023-10-30T12:10:20.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe29ffff640 15 client.4247.objecter _session_command_op_remove 1 2 2023-10-30T12:10:20.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 10 librados: watch_flush enter 2023-10-30T12:10:20.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 10 librados: watch_flush exit 2023-10-30T12:10:20.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 10 client.4247.objecter close_session for osd.1 2023-10-30T12:10:20.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 1 -- 172.21.15.3:0/2306484969 >> v2:172.21.15.3:6804/1445650280 conn(0x7fe2a4072620 msgr2=0x7fe2a4074a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 1 --2- 172.21.15.3:0/2306484969 >> v2:172.21.15.3:6804/1445650280 conn(0x7fe2a4072620 0x7fe2a4074a40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 20 client.4247.objecter put_session s=0x7fe2b0003bb0 osd=1 1 2023-10-30T12:10:21.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 20 client.4247.objecter shutdown clearing up homeless session... 2023-10-30T12:10:21.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 10 client.4247.objecter successfully canceled tick 2023-10-30T12:10:21.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 1 -- 172.21.15.3:0/2306484969 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe2a406c810 msgr2=0x7fe2a406ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 1 --2- 172.21.15.3:0/2306484969 >> v2:172.21.15.92:6800/1750155735 conn(0x7fe2a406c810 0x7fe2a406ecd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe2a80017f0 tx=0x7fe2a8009040 comp rx=0 tx=0).stop 2023-10-30T12:10:21.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 10 monclient: shutdown 2023-10-30T12:10:21.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:21.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 1 -- 172.21.15.3:0/2306484969 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2b8142d60 msgr2=0x7fe2b81431f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 1 --2- 172.21.15.3:0/2306484969 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2b8142d60 0x7fe2b81431f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000cc60 tx=0x7fe2b000a5c0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2b7fff640 1 -- 172.21.15.3:0/2306484969 reap_dead start 2023-10-30T12:10:21.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 1 -- 172.21.15.3:0/2306484969 shutdown_connections 2023-10-30T12:10:21.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 1 -- 172.21.15.3:0/2306484969 >> 172.21.15.3:0/2306484969 conn(0x7fe2b80b7180 msgr2=0x7fe2b80b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:21.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 1 -- 172.21.15.3:0/2306484969 shutdown_connections 2023-10-30T12:10:21.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 1 -- 172.21.15.3:0/2306484969 wait complete. 2023-10-30T12:10:21.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.726+0000 7fe2bdd65640 1 librados: shutdown 2023-10-30T12:10:21.003 INFO:teuthology.orchestra.run.smithi003.stdout:60129542149 2023-10-30T12:10:21.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T12:10:21.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.740+0000 7f367bfff640 1 -- 172.21.15.3:0/3670555450 <== osd.0 v2:172.21.15.3:6808/3639283465 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f36840052d0 con 0x7f3680072620 2023-10-30T12:10:21.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.740+0000 7f367bfff640 10 client.4256.objecter ms_dispatch 0x7f3694000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:10:21.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.740+0000 7f367bfff640 10 client.4256.objecter _finish_command 1 = system:0 2023-10-30T12:10:21.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.740+0000 7f367bfff640 20 client.4256.objecter put_session s=0x7f3684003830 osd=0 4 2023-10-30T12:10:21.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.740+0000 7f367bfff640 15 client.4256.objecter _session_command_op_remove 0 1 2023-10-30T12:10:21.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.758+0000 7f3699ffb640 10 client.4256.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:10:21.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.758+0000 7f3699ffb640 15 client.4256.objecter _session_command_op_assign -1 2 2023-10-30T12:10:21.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.758+0000 7f3699ffb640 20 client.4256.objecter _get_session s=0x7f3684003830 osd=0 3 2023-10-30T12:10:21.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.758+0000 7f3699ffb640 20 client.4256.objecter put_session s=0x7f3684003830 osd=0 3 2023-10-30T12:10:21.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.758+0000 7f3699ffb640 20 client.4256.objecter _get_session s=0x7f3684003830 osd=0 3 2023-10-30T12:10:21.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.758+0000 7f3699ffb640 15 client.4256.objecter _session_command_op_remove -1 2 2023-10-30T12:10:21.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.758+0000 7f3699ffb640 20 client.4256.objecter get_session s=0x7f3684003830 osd=0 3 2023-10-30T12:10:21.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.758+0000 7f3699ffb640 15 client.4256.objecter _session_command_op_assign 0 2 2023-10-30T12:10:21.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.758+0000 7f3699ffb640 20 client.4256.objecter put_session s=0x7f3684003830 osd=0 4 2023-10-30T12:10:21.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.758+0000 7f3699ffb640 10 client.4256.objecter _send_command 2 2023-10-30T12:10:21.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.758+0000 7f3699ffb640 1 -- 172.21.15.3:0/3670555450 --> v2:172.21.15.3:6808/3639283465 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f36940b1f50 con 0x7f3680072620 2023-10-30T12:10:21.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.763+0000 7f367bfff640 1 -- 172.21.15.3:0/3670555450 <== osd.0 v2:172.21.15.3:6808/3639283465 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f368c034440 con 0x7f3680072620 2023-10-30T12:10:21.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.763+0000 7f367bfff640 10 client.4256.objecter ms_dispatch 0x7f3694000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:10:21.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.763+0000 7f367bfff640 10 client.4256.objecter _finish_command 2 = system:0 2023-10-30T12:10:21.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.763+0000 7f367bfff640 20 client.4256.objecter put_session s=0x7f3684003830 osd=0 4 2023-10-30T12:10:21.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.763+0000 7f367bfff640 15 client.4256.objecter _session_command_op_remove 0 2 2023-10-30T12:10:21.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 10 librados: watch_flush enter 2023-10-30T12:10:21.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 10 librados: watch_flush exit 2023-10-30T12:10:21.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 10 client.4256.objecter close_session for osd.0 2023-10-30T12:10:21.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 1 -- 172.21.15.3:0/3670555450 >> v2:172.21.15.3:6808/3639283465 conn(0x7f3680072620 msgr2=0x7f3680074a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 1 --2- 172.21.15.3:0/3670555450 >> v2:172.21.15.3:6808/3639283465 conn(0x7f3680072620 0x7f3680074a40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 20 client.4256.objecter put_session s=0x7f3684003830 osd=0 1 2023-10-30T12:10:21.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 20 client.4256.objecter shutdown clearing up homeless session... 2023-10-30T12:10:21.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 10 client.4256.objecter successfully canceled tick 2023-10-30T12:10:21.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 1 -- 172.21.15.3:0/3670555450 >> v2:172.21.15.92:6800/1750155735 conn(0x7f368006c810 msgr2=0x7f368006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 1 --2- 172.21.15.3:0/3670555450 >> v2:172.21.15.92:6800/1750155735 conn(0x7f368006c810 0x7f368006ecd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f36940b3ee0 tx=0x7f3688009040 comp rx=0 tx=0).stop 2023-10-30T12:10:21.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 10 monclient: shutdown 2023-10-30T12:10:21.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:21.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 1 -- 172.21.15.3:0/3670555450 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f36940a88d0 msgr2=0x7f369413eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 1 --2- 172.21.15.3:0/3670555450 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f36940a88d0 0x7f369413eaf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f368400c700 tx=0x7f368400cbc0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3693fff640 1 -- 172.21.15.3:0/3670555450 reap_dead start 2023-10-30T12:10:21.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 1 -- 172.21.15.3:0/3670555450 shutdown_connections 2023-10-30T12:10:21.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 1 -- 172.21.15.3:0/3670555450 >> 172.21.15.3:0/3670555450 conn(0x7f36940b7180 msgr2=0x7f3694142ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:21.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 1 -- 172.21.15.3:0/3670555450 shutdown_connections 2023-10-30T12:10:21.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 1 -- 172.21.15.3:0/3670555450 wait complete. 2023-10-30T12:10:21.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:20.764+0000 7f3699ffb640 1 librados: shutdown 2023-10-30T12:10:21.015 INFO:teuthology.orchestra.run.smithi003.stdout:68719476741 2023-10-30T12:10:21.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T12:10:21.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.212+0000 7f3de4f05640 10 monclient: get_monmap_and_config 2023-10-30T12:10:21.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.212+0000 7f3de4f05640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.212+0000 7f3de4f05640 10 monclient: monmap: 2023-10-30T12:10:21.215 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:21.215 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.215 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:21.213886+0000 2023-10-30T12:10:21.215 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:21.213886+0000 2023-10-30T12:10:21.216 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.216 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.216 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:21.216 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:21.216 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:21.216 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:21.217 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:21.217 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.213+0000 7ff7ddf5a640 10 monclient: get_monmap_and_config 2023-10-30T12:10:21.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.213+0000 7ff7ddf5a640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.213+0000 7ff7ddf5a640 10 monclient: monmap: 2023-10-30T12:10:21.218 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:21.218 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.219 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:21.214285+0000 2023-10-30T12:10:21.219 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:21.214285+0000 2023-10-30T12:10:21.219 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.219 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.219 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:21.219 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:21.220 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:21.220 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:21.220 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:21.220 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.214+0000 7ff7ddf5a640 1 Processor -- start 2023-10-30T12:10:21.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.214+0000 7ff7ddf5a640 1 -- start start 2023-10-30T12:10:21.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.214+0000 7ff7ddf5a640 10 monclient: init 2023-10-30T12:10:21.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.214+0000 7f3de4f05640 1 Processor -- start 2023-10-30T12:10:21.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.215+0000 7ff7ddf5a640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.215+0000 7ff7ddf5a640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-10-30T12:10:21.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.215+0000 7ff7ddf5a640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff7d80b0990 0x7ff7d8071cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.215+0000 7ff7ddf5a640 10 monclient(hunting): picked mon.noname-c con 0x7ff7d80b0990 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:21.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.215+0000 7ff7ddf5a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff7d806eea0 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:21.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.215+0000 7ff7ddf5a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff7d806f2c0 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:21.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.215+0000 7ff7ddf5a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d8070980 con 0x7ff7d806f2c0 2023-10-30T12:10:21.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.215+0000 7ff7ddf5a640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d8070c60 con 0x7ff7d806eea0 2023-10-30T12:10:21.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.215+0000 7ff7ddf5a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.215+0000 7ff7ddf5a640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d80679c0 con 0x7ff7d80b0990 2023-10-30T12:10:21.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.215+0000 7ff7ddf5a640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.215+0000 7ff7ddf5a640 10 monclient(hunting): authenticate will time out at 1412.429523s 2023-10-30T12:10:21.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.214+0000 7f3de4f05640 1 -- start start 2023-10-30T12:10:21.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.214+0000 7f3de4f05640 10 monclient: init 2023-10-30T12:10:21.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.215+0000 7f63d4026640 10 monclient: get_monmap_and_config 2023-10-30T12:10:21.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.215+0000 7f63d4026640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.215+0000 7f63d4026640 10 monclient: monmap: 2023-10-30T12:10:21.226 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:21.226 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.226 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:21.216831+0000 2023-10-30T12:10:21.227 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:21.216831+0000 2023-10-30T12:10:21.227 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.227 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.227 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:21.227 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:21.228 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:21.229 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:21.229 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:21.229 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7f3de4f05640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7f3de4f05640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2023-10-30T12:10:21.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7f3de4f05640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3de0071180 0x7f3de0071560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7f3de4f05640 10 monclient(hunting): picked mon.noname-b con 0x7f3de0071180 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:21.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7f3de4f05640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3de0141b30 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:21.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7f3de4f05640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3de0141f50 0x7f3de006f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7f3de4f05640 10 monclient(hunting): picked mon.noname-a con 0x7f3de0141f50 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:21.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7f3de4f05640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de01412c0 con 0x7f3de0141b30 2023-10-30T12:10:21.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7f3de4f05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7f3de4f05640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3de0067a30 con 0x7f3de0141f50 2023-10-30T12:10:21.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7f3de4f05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7f3de4f05640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de00706a0 con 0x7f3de0071180 2023-10-30T12:10:21.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7f3de4f05640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7f3de4f05640 10 monclient(hunting): authenticate will time out at 1412.430200s 2023-10-30T12:10:21.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7ff7d7fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff7d806f2c0 0x7ff7d8148600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50422/0 (socket says 172.21.15.3:50422) 2023-10-30T12:10:21.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.216+0000 7ff7d7fff640 1 -- 172.21.15.3:0/490902170 learned_addr learned my addr 172.21.15.3:0/490902170 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:21.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.217+0000 7f3dde59c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3de0071180 0x7f3de0071560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.217+0000 7ff7d77fe640 1 --2- 172.21.15.3:0/490902170 >> v2:172.21.15.92:3300/0 conn(0x7ff7d80b0990 0x7ff7d8071cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.217+0000 7ff7d77fe640 10 monclient(hunting): get_auth_request con 0x7ff7d80b0990 auth_method 0 2023-10-30T12:10:21.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.217+0000 7ff7d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.217+0000 7ff7d77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.217+0000 7ff7d77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.217+0000 7f3dde59c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3de0071180 0x7f3de0071560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:60836/0 (socket says 172.21.15.3:60836) 2023-10-30T12:10:21.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.217+0000 7f3dde59c640 1 -- 172.21.15.3:0/390127575 learned_addr learned my addr 172.21.15.3:0/390127575 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:21.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.217+0000 7f3dde59c640 10 monclient(hunting): get_auth_request con 0x7f3de0071180 auth_method 0 2023-10-30T12:10:21.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.217+0000 7f3dde59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.217+0000 7f3dde59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.217+0000 7f3dde59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.218+0000 7f9c067b0640 10 monclient: get_monmap_and_config 2023-10-30T12:10:21.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.218+0000 7f9c067b0640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.218+0000 7f9c067b0640 10 monclient: monmap: 2023-10-30T12:10:21.237 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:21.237 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.237 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:21.219419+0000 2023-10-30T12:10:21.238 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:21.219419+0000 2023-10-30T12:10:21.238 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.238 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.238 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:21.238 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:21.239 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:21.239 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:21.239 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:21.239 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.218+0000 7ff7d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.218+0000 7ff7d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.218+0000 7ff7d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.218+0000 7f3dde59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.218+0000 7f3dde59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.218+0000 7f3dde59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.219+0000 7f63d4026640 1 Processor -- start 2023-10-30T12:10:21.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.219+0000 7f63d4026640 1 -- start start 2023-10-30T12:10:21.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.219+0000 7f63d4026640 10 monclient: init 2023-10-30T12:10:21.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.219+0000 7fb8870df640 10 monclient: get_monmap_and_config 2023-10-30T12:10:21.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.219+0000 7fb8870df640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.219+0000 7fb8870df640 10 monclient: monmap: 2023-10-30T12:10:21.243 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:21.243 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.243 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:21.220418+0000 2023-10-30T12:10:21.244 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:21.220418+0000 2023-10-30T12:10:21.244 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.244 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.245 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:21.245 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:21.245 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:21.245 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:21.245 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:21.246 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7fb8870df640 1 Processor -- start 2023-10-30T12:10:21.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7ff7d67fc640 1 -- 172.21.15.3:0/490902170 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692483836 0 0) 0x7ff7c8001110 con 0x7ff7d806f2c0 2023-10-30T12:10:21.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7ff7d67fc640 10 monclient(hunting): _negotiate 2023-10-30T12:10:21.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7ff7d67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7ff7d67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7ff7d67fc640 10 monclient(hunting): my global_id is 4338 2023-10-30T12:10:21.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7ff7d67fc640 1 -- 172.21.15.3:0/490902170 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7c0003520 con 0x7ff7d806f2c0 2023-10-30T12:10:21.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7f63d4026640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7f63d4026640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2023-10-30T12:10:21.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7f63d4026640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f63c4038ef0 0x7f63c40c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7f63d4026640 10 monclient(hunting): picked mon.noname-a con 0x7f63c4038ef0 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:21.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7f63d4026640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c40392d0 0x7f63c40c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7f63d4026640 10 monclient(hunting): picked mon.noname-b con 0x7f63c40392d0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:21.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7f63d4026640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63c40c8310 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:21.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7f63d4026640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c40374a0 con 0x7f63c40c8310 2023-10-30T12:10:21.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7f63d4026640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7f63d4026640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c4008750 con 0x7f63c4038ef0 2023-10-30T12:10:21.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7f63d4026640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7f63d4026640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4037780 con 0x7f63c40392d0 2023-10-30T12:10:21.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7f63d4026640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7f63d4026640 10 monclient(hunting): authenticate will time out at 1412.435146s 2023-10-30T12:10:21.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7fb8870df640 1 -- start start 2023-10-30T12:10:21.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.220+0000 7fb8870df640 10 monclient: init 2023-10-30T12:10:21.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7ff9bd42d640 10 monclient: get_monmap_and_config 2023-10-30T12:10:21.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7ff9bd42d640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7ff9bd42d640 10 monclient: monmap: 2023-10-30T12:10:21.253 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:21.253 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.254 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:21.222313+0000 2023-10-30T12:10:21.254 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:21.222313+0000 2023-10-30T12:10:21.254 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.254 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:21.255 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7f9d91a94640 10 monclient: get_monmap_and_config 2023-10-30T12:10:21.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7fb8870df640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7fb8870df640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-10-30T12:10:21.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7fb8870df640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb878034aa0 0x7fb87802a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7fb8870df640 10 monclient(hunting): picked mon.noname-b con 0x7fb878034aa0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:21.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7fb8870df640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb878034e80 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:21.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7fb8870df640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb8780c3d00 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7fb8870df640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb878032f60 con 0x7fb8780c3d00 2023-10-30T12:10:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7fb8870df640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87802cd10 con 0x7fb878034e80 2023-10-30T12:10:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7fb8870df640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7fb8870df640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb878008720 con 0x7fb878034aa0 2023-10-30T12:10:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7fb8870df640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7fb8870df640 10 monclient(hunting): authenticate will time out at 1412.435736s 2023-10-30T12:10:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7f9d91a94640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7f9d91a94640 10 monclient: monmap: 2023-10-30T12:10:21.260 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:21.260 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.261 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:21.222569+0000 2023-10-30T12:10:21.261 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:21.222569+0000 2023-10-30T12:10:21.261 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.261 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.261 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:21.262 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:21.262 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:21.262 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:21.262 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:21.262 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7ff9bd42d640 1 Processor -- start 2023-10-30T12:10:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7f3dde59c640 10 monclient(hunting): handle_auth_done global_id 4259 payload 274 2023-10-30T12:10:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7f3dde59c640 1 -- 172.21.15.3:0/390127575 >> v2:172.21.15.3:3300/0 conn(0x7f3de0141f50 msgr2=0x7f3de006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7f3dde59c640 1 --2- 172.21.15.3:0/390127575 >> v2:172.21.15.3:3300/0 conn(0x7f3de0141f50 0x7f3de006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7f3dde59c640 1 -- 172.21.15.3:0/390127575 >> v1:172.21.15.3:6789/0 conn(0x7f3de0141b30 legacy=0x7f3de006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7f3dde59c640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7f3dde59c640 1 monclient: found mon.noname-b 2023-10-30T12:10:21.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7f3dde59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7f3dde59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:10:21.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7f3dde59c640 1 -- 172.21.15.3:0/390127575 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de0070980 con 0x7f3de0071180 2023-10-30T12:10:21.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7f9c067b0640 1 Processor -- start 2023-10-30T12:10:21.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7f9c067b0640 1 -- start start 2023-10-30T12:10:21.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7f9c067b0640 10 monclient: init 2023-10-30T12:10:21.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.221+0000 7ff7d77fe640 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2023-10-30T12:10:21.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7ff7d77fe640 1 -- 172.21.15.3:0/490902170 >> v1:172.21.15.92:6789/0 conn(0x7ff7d806eea0 legacy=0x7ff7d813af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7ff7d77fe640 1 -- 172.21.15.3:0/490902170 >> v1:172.21.15.3:6789/0 conn(0x7ff7d806f2c0 legacy=0x7ff7d8148600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7ff7d77fe640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7ff7d77fe640 1 monclient: found mon.noname-c 2023-10-30T12:10:21.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7ff7d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7ff7d77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7ff7d77fe640 1 -- 172.21.15.3:0/490902170 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d80717d0 con 0x7ff7d80b0990 2023-10-30T12:10:21.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7f9c067b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9c067b0640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-10-30T12:10:21.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9c067b0640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9bf80070a0 0x7f9bf80338a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9c067b0640 10 monclient(hunting): picked mon.noname-a con 0x7f9bf80070a0 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:21.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9c067b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9bf80307d0 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:21.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7fb8860dd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb878034aa0 0x7fb87802a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9c067b0640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9bf8036280 0x7f9bf80d68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9c067b0640 10 monclient(hunting): picked mon.noname-c con 0x7f9bf8036280 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:21.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9c067b0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf80375f0 con 0x7f9bf80307d0 2023-10-30T12:10:21.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9c067b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9c067b0640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf8008610 con 0x7f9bf80070a0 2023-10-30T12:10:21.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9c067b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9c067b0640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf8007480 con 0x7f9bf8036280 2023-10-30T12:10:21.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9c067b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9c067b0640 10 monclient(hunting): authenticate will time out at 1412.437165s 2023-10-30T12:10:21.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7fb8860dd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb878034aa0 0x7fb87802a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:60848/0 (socket says 172.21.15.3:60848) 2023-10-30T12:10:21.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7fb8860dd640 1 -- 172.21.15.3:0/904257447 learned_addr learned my addr 172.21.15.3:0/904257447 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:21.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7fb8860dd640 10 monclient(hunting): get_auth_request con 0x7fb878034aa0 auth_method 0 2023-10-30T12:10:21.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7fb8860dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7fb8860dd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7fb8860dd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7ff9bd42d640 1 -- start start 2023-10-30T12:10:21.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.222+0000 7ff9bd42d640 10 monclient: init 2023-10-30T12:10:21.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-10-30T12:10:21.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff9b80b4350 0x7ff9b80a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9d91a94640 1 Processor -- start 2023-10-30T12:10:21.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f63d25c3640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f63c40c8310 0x7f63c40cb350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.3:46048/0 (socket says 172.21.15.3:46048) 2023-10-30T12:10:21.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f63d25c3640 1 -- 172.21.15.3:0/2349584406 learned_addr learned my addr 172.21.15.3:0/2349584406 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:21.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 10 monclient(hunting): picked mon.noname-a con 0x7ff9b80b4350 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:21.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9b813eb50 0x7ff9b813ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 10 monclient(hunting): picked mon.noname-b con 0x7ff9b813eb50 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:21.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff9b8144f50 0x7ff9b81453e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 10 monclient(hunting): picked mon.noname-c con 0x7ff9b8144f50 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:21.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f63d15c1640 1 --2- 172.21.15.3:0/2349584406 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c40392d0 0x7f63c40c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b80679f0 con 0x7ff9b80b4350 2023-10-30T12:10:21.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b80b4730 con 0x7ff9b8144f50 2023-10-30T12:10:21.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b80aa6e0 con 0x7ff9b813eb50 2023-10-30T12:10:21.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7ff9bd42d640 10 monclient(hunting): authenticate will time out at 1412.437664s 2023-10-30T12:10:21.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9d91a94640 1 -- start start 2023-10-30T12:10:21.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.223+0000 7f9d91a94640 10 monclient: init 2023-10-30T12:10:21.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7fb8860dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7fb8860dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7fb8860dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f9d91a94640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f9d91a94640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-10-30T12:10:21.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f9d91a94640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9d8c0b4350 0x7f9d8c0a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f9d91a94640 10 monclient(hunting): picked mon.noname-a con 0x7f9d8c0b4350 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:21.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f9d91a94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d8c13eb50 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:21.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f9d91a94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d8c144f50 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:21.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f9d91a94640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c13f6b0 con 0x7f9d8c144f50 2023-10-30T12:10:21.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f9d91a94640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c0aa6e0 con 0x7f9d8c13eb50 2023-10-30T12:10:21.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f9d91a94640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f9d91a94640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8c0679f0 con 0x7f9d8c0b4350 2023-10-30T12:10:21.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f9d91a94640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f9d91a94640 10 monclient(hunting): authenticate will time out at 1412.438633s 2023-10-30T12:10:21.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f63d15c1640 10 monclient(hunting): get_auth_request con 0x7f63c40392d0 auth_method 0 2023-10-30T12:10:21.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f63d15c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f63d15c1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.224+0000 7f63d15c1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.226+0000 7fb8860dd640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2023-10-30T12:10:21.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.226+0000 7fb8860dd640 1 -- 172.21.15.3:0/904257447 >> v1:172.21.15.92:6789/0 conn(0x7fb878034e80 legacy=0x7fb878032850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.226+0000 7fb8860dd640 1 -- 172.21.15.3:0/904257447 >> v1:172.21.15.3:6789/0 conn(0x7fb8780c3d00 legacy=0x7fb8780ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:21.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.226+0000 7fb8860dd640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.226+0000 7fb8860dd640 1 monclient: found mon.noname-b 2023-10-30T12:10:21.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.226+0000 7fb8860dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.226+0000 7fb8860dd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:10:21.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.226+0000 7fb8860dd640 1 -- 172.21.15.3:0/904257447 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8780c1840 con 0x7fb878034aa0 2023-10-30T12:10:21.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.227+0000 7fbbfc997640 10 monclient: get_monmap_and_config 2023-10-30T12:10:21.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.227+0000 7fbbfc997640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.227+0000 7fbbfc997640 10 monclient: monmap: 2023-10-30T12:10:21.289 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:21.289 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.289 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:21.228304+0000 2023-10-30T12:10:21.289 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:21.228304+0000 2023-10-30T12:10:21.289 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.290 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.290 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:21.290 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:21.290 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:21.290 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:21.291 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:21.291 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.227+0000 7f63d15c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.227+0000 7f63d15c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.227+0000 7f63d15c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.227+0000 7fbbfc997640 1 Processor -- start 2023-10-30T12:10:21.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.228+0000 7fbbfc997640 1 -- start start 2023-10-30T12:10:21.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.228+0000 7fbbfc997640 10 monclient: init 2023-10-30T12:10:21.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.228+0000 7f9d8affd640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f9d8c13eb50 0x7f9d8c13efa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.3:46068/0 (socket says 172.21.15.3:46068) 2023-10-30T12:10:21.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.228+0000 7f9d8affd640 1 -- 172.21.15.3:0/2555943919 learned_addr learned my addr 172.21.15.3:0/2555943919 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:21.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.228+0000 7fbbfc997640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.228+0000 7fbbfc997640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2023-10-30T12:10:21.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.229+0000 7fbbfc997640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbbf80a8e40 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:21.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.229+0000 7fbbfc997640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf80b13c0 0x7fbbf8141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.229+0000 7fbbfc997640 10 monclient(hunting): picked mon.noname-b con 0x7fbbf80b13c0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:21.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.229+0000 7fbbfc997640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbbf81423f0 0x7fbbf814c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.229+0000 7f9c04d4d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9bf8036280 0x7f9bf80d68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.229+0000 7fbbfc997640 10 monclient(hunting): picked mon.noname-c con 0x7fbbf81423f0 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:21.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.229+0000 7fbbfc997640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf80b1b00 con 0x7fbbf80a8e40 2023-10-30T12:10:21.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.229+0000 7fbbfc997640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.229+0000 7fbbfc997640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf80679a0 con 0x7fbbf81423f0 2023-10-30T12:10:21.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.229+0000 7fbbfc997640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.229+0000 7fbbfc997640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf80b1060 con 0x7fbbf80b13c0 2023-10-30T12:10:21.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.229+0000 7fbbfc997640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.229+0000 7fbbfc997640 10 monclient(hunting): authenticate will time out at 1412.443423s 2023-10-30T12:10:21.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.230+0000 7f9c04d4d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9bf8036280 0x7f9bf80d68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56334/0 (socket says 172.21.15.3:56334) 2023-10-30T12:10:21.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.230+0000 7f9c04d4d640 1 -- 172.21.15.3:0/2589113220 learned_addr learned my addr 172.21.15.3:0/2589113220 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:21.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.230+0000 7f9c04d4d640 10 monclient(hunting): get_auth_request con 0x7f9bf8036280 auth_method 0 2023-10-30T12:10:21.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.230+0000 7f9c04d4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.230+0000 7f9c04d4d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.230+0000 7f9c04d4d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.230+0000 7ff7d77fe640 1 --2- 172.21.15.3:0/490902170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7d80b0990 0x7ff7d8071cf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff7d000d050 tx=0x7ff7d002e140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6878425d3cba7949 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.230+0000 7f9bfffff640 1 --2- 172.21.15.3:0/2589113220 >> v2:172.21.15.3:3300/0 conn(0x7f9bf80070a0 0x7f9bf80338a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.230+0000 7ff9b6ffd640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff9b80b4350 0x7ff9b80a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.230+0000 7f9c04d4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.230+0000 7f9c04d4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.230+0000 7f9c04d4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.231+0000 7f9bfffff640 10 monclient(hunting): get_auth_request con 0x7f9bf80070a0 auth_method 0 2023-10-30T12:10:21.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.231+0000 7f9bfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.231+0000 7f9bfffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.231+0000 7f9bfffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.231+0000 7ff9b6ffd640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff9b80b4350 0x7ff9b80a83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42354/0 (socket says 172.21.15.3:42354) 2023-10-30T12:10:21.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.231+0000 7ff9b6ffd640 1 -- 172.21.15.3:0/2018086619 learned_addr learned my addr 172.21.15.3:0/2018086619 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:21.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.231+0000 7ff9b6ffd640 10 monclient(hunting): get_auth_request con 0x7ff9b80b4350 auth_method 0 2023-10-30T12:10:21.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.231+0000 7ff9b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.231+0000 7ff9b6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.231+0000 7ff9b6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.232+0000 7f63d15c1640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2023-10-30T12:10:21.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.232+0000 7f63d15c1640 1 -- 172.21.15.3:0/2349584406 >> v2:172.21.15.3:3300/0 conn(0x7f63c4038ef0 msgr2=0x7f63c40c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:21.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.232+0000 7f63d15c1640 1 --2- 172.21.15.3:0/2349584406 >> v2:172.21.15.3:3300/0 conn(0x7f63c4038ef0 0x7f63c40c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.232+0000 7f63d15c1640 1 -- 172.21.15.3:0/2349584406 >> v1:172.21.15.92:6789/0 conn(0x7f63c40c8310 legacy=0x7f63c40cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.232+0000 7f63d15c1640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.232+0000 7f63d15c1640 1 monclient: found mon.noname-b 2023-10-30T12:10:21.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.232+0000 7f63d15c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.232+0000 7f63d15c1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:10:21.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.232+0000 7f63d15c1640 1 -- 172.21.15.3:0/2349584406 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c4036590 con 0x7f63c40392d0 2023-10-30T12:10:21.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.232+0000 7f9bfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.232+0000 7f9bfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.232+0000 7f9bfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.233+0000 7ff9b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.233+0000 7ff9b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.233+0000 7f3dde59c640 1 --2- 172.21.15.3:0/390127575 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3de0071180 0x7f3de0071560 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4009870 tx=0x7f3dd402b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fd81ac44a2aac9e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.233+0000 7ff9b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.233+0000 7fb8860dd640 1 --2- 172.21.15.3:0/904257447 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb878034aa0 0x7fb87802a950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009210 tx=0x7fb87c02cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cbdadcb7ec41084 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7f9c04d4d640 10 monclient(hunting): handle_auth_done global_id 4324 payload 274 2023-10-30T12:10:21.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7f9c04d4d640 1 -- 172.21.15.3:0/2589113220 >> v2:172.21.15.3:3300/0 conn(0x7f9bf80070a0 msgr2=0x7f9bf80338a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:21.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7f9c04d4d640 1 --2- 172.21.15.3:0/2589113220 >> v2:172.21.15.3:3300/0 conn(0x7f9bf80070a0 0x7f9bf80338a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7f9c04d4d640 1 -- 172.21.15.3:0/2589113220 >> v1:172.21.15.3:6789/0 conn(0x7f9bf80307d0 legacy=0x7f9bf8035b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:21.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7f9c04d4d640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7f9c04d4d640 1 monclient: found mon.noname-c 2023-10-30T12:10:21.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7f9c04d4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7f9c04d4d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7f9c04d4d640 1 -- 172.21.15.3:0/2589113220 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf8030c60 con 0x7f9bf8036280 2023-10-30T12:10:21.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7ff9b77fe640 1 --2- 172.21.15.3:0/2018086619 >> v2:172.21.15.92:3300/0 conn(0x7ff9b8144f50 0x7ff9b81453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7ff9b6ffd640 10 monclient(hunting): handle_auth_done global_id 4344 payload 274 2023-10-30T12:10:21.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7ff9b6ffd640 1 -- 172.21.15.3:0/2018086619 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9b813eb50 msgr2=0x7ff9b813ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:21.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7ff9b6ffd640 1 --2- 172.21.15.3:0/2018086619 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9b813eb50 0x7ff9b813ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7ff9b6ffd640 1 -- 172.21.15.3:0/2018086619 >> v2:172.21.15.92:3300/0 conn(0x7ff9b8144f50 msgr2=0x7ff9b81453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:21.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7ff9b6ffd640 1 --2- 172.21.15.3:0/2018086619 >> v2:172.21.15.92:3300/0 conn(0x7ff9b8144f50 0x7ff9b81453e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7ff9b6ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7ff9b6ffd640 1 monclient: found mon.noname-a 2023-10-30T12:10:21.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7ff9b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7ff9b6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:21.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.234+0000 7ff9b6ffd640 1 -- 172.21.15.3:0/2018086619 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b80a8910 con 0x7ff9b80b4350 2023-10-30T12:10:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.235+0000 7f9d8a7fc640 1 -- 172.21.15.3:0/2555943919 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307284729 0 0) 0x7f9d7c001110 con 0x7f9d8c13eb50 2023-10-30T12:10:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.235+0000 7f9d8a7fc640 10 monclient(hunting): _negotiate 2023-10-30T12:10:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.235+0000 7f9d8a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.235+0000 7f9d8a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.235+0000 7f9d8a7fc640 10 monclient(hunting): my global_id is 4327 2023-10-30T12:10:21.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.235+0000 7f9d8a7fc640 1 -- 172.21.15.3:0/2555943919 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d74003520 con 0x7f9d8c13eb50 2023-10-30T12:10:21.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.236+0000 7f63d15c1640 1 --2- 172.21.15.3:0/2349584406 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c40392d0 0x7f63c40c7dd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0097a0 tx=0x7f63bc02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69c3ac4c33bc66b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 1 -- 172.21.15.3:0/2555943919 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2973328117 0 0) 0x7f9d7c001510 con 0x7f9d8c13eb50 2023-10-30T12:10:21.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 1 -- 172.21.15.3:0/2555943919 >> v2:172.21.15.3:3300/0 conn(0x7f9d8c0b4350 msgr2=0x7f9d8c0a83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:21.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 1 --2- 172.21.15.3:0/2555943919 >> v2:172.21.15.3:3300/0 conn(0x7f9d8c0b4350 0x7f9d8c0a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 1 -- 172.21.15.3:0/2555943919 >> v1:172.21.15.3:6789/0 conn(0x7f9d8c144f50 legacy=0x7f9d8c1453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:21.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7ff9b6ffd640 1 --2- 172.21.15.3:0/2018086619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b80b4350 0x7ff9b80a83d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff9b80a9770 tx=0x7ff9a802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e23491624a5c00 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:10:21.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.92:6789/0 2023-10-30T12:10:21.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 1 -- 172.21.15.3:0/2555943919 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c0a8910 con 0x7f9d8c13eb50 2023-10-30T12:10:21.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d91a94640 5 monclient: authenticate success, global_id 4327 2023-10-30T12:10:21.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d91a94640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:21.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 1 -- 172.21.15.3:0/2555943919 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1777801622 0 0) 0x7f9d7c0019b0 con 0x7f9d8c13eb50 2023-10-30T12:10:21.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:10:21.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient: dump: 2023-10-30T12:10:21.321 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.321 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.322 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.322 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.322 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.322 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.322 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.323 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.323 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.323 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] but i'm connected to v1:172.21.15.92:6789/0, reconnecting 2023-10-30T12:10:21.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 1 -- 172.21.15.3:0/2555943919 >> v1:172.21.15.92:6789/0 conn(0x7f9d8c13eb50 legacy=0x7f9d8c13efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:10:21.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 1 --2- 172.21.15.3:0/2555943919 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d740039c0 0x7f9d74003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient(hunting): picked mon.c con 0x7f9d740039c0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:21.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 1 --2- 172.21.15.3:0/2555943919 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d740047c0 0x7f9d74004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient(hunting): picked mon.a con 0x7f9d740047c0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:21.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 1 --2- 172.21.15.3:0/2555943919 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d740053d0 0x7f9d7400ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient(hunting): picked mon.b con 0x7f9d740053d0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:21.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 1 -- 172.21.15.3:0/2555943919 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0679f0 con 0x7f9d740047c0 2023-10-30T12:10:21.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 1 -- 172.21.15.3:0/2555943919 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7400c310 con 0x7f9d740053d0 2023-10-30T12:10:21.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 1 -- 172.21.15.3:0/2555943919 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7400c4e0 con 0x7f9d740039c0 2023-10-30T12:10:21.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d8a7fc640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:10:21.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.240+0000 7f9d91a94640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8850db640 1 -- 172.21.15.3:0/904257447 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c02d940 con 0x7fb878034aa0 2023-10-30T12:10:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8850db640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8850db640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:21.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8850db640 10 monclient: dump: 2023-10-30T12:10:21.330 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.330 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.330 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.331 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.331 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.331 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.331 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.331 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.332 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.332 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8850db640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8850db640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8850db640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8870df640 5 monclient: authenticate success, global_id 4262 2023-10-30T12:10:21.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8870df640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:21.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8850db640 1 -- 172.21.15.3:0/904257447 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c02dab0 con 0x7fb878034aa0 2023-10-30T12:10:21.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8850db640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8870df640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:21.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8870df640 4 set_mon_vals no callback set 2023-10-30T12:10:21.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8850db640 1 -- 172.21.15.3:0/904257447 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c02de50 con 0x7fb878034aa0 2023-10-30T12:10:21.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8850db640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8850db640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:21.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.241+0000 7fb8850db640 10 monclient: dump: 2023-10-30T12:10:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.335 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.336 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.336 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.336 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.336 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.336 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.242+0000 7fb8870df640 10 monclient: shutdown 2023-10-30T12:10:21.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.242+0000 7fb8870df640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:21.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.242+0000 7fb8870df640 1 -- 172.21.15.3:0/904257447 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb878034aa0 msgr2=0x7fb87802a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.242+0000 7fb8870df640 1 --2- 172.21.15.3:0/904257447 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb878034aa0 0x7fb87802a950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009210 tx=0x7fb87c02cb90 comp rx=0 tx=0).stop 2023-10-30T12:10:21.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.242+0000 7fb8870df640 1 -- 172.21.15.3:0/904257447 shutdown_connections 2023-10-30T12:10:21.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.242+0000 7fb8870df640 1 --2- 172.21.15.3:0/904257447 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb878034aa0 0x7fb87802a950 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.242+0000 7fb8870df640 1 -- 172.21.15.3:0/904257447 >> 172.21.15.3:0/904257447 conn(0x7fb8780060a0 msgr2=0x7fb8780064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:21.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.242+0000 7fb8870df640 1 -- 172.21.15.3:0/904257447 shutdown_connections 2023-10-30T12:10:21.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.242+0000 7fb8870df640 1 -- 172.21.15.3:0/904257447 wait complete. 2023-10-30T12:10:21.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.243+0000 7fb8870df640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.243+0000 7fb8870df640 10 monclient: monmap: 2023-10-30T12:10:21.339 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:21.339 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:21.339 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:21.340 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:21.340 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.340 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:21.340 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:21.340 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:21.340 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:21.341 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.243+0000 7fb8870df640 1 librados: starting msgr at 2023-10-30T12:10:21.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.243+0000 7fb8870df640 1 librados: starting objecter 2023-10-30T12:10:21.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.243+0000 7fb8870df640 1 Processor -- start 2023-10-30T12:10:21.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.243+0000 7fb8870df640 1 -- start start 2023-10-30T12:10:21.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.243+0000 7fb8870df640 1 librados: setting wanted keys 2023-10-30T12:10:21.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.243+0000 7fb8870df640 1 librados: calling monclient init 2023-10-30T12:10:21.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.243+0000 7fb8870df640 10 monclient: init 2023-10-30T12:10:21.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:10:21.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7f9c04d4d640 1 --2- 172.21.15.3:0/2589113220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf8036280 0x7f9bf80d68e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9bf80cd130 tx=0x7f9bec02b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb6171cce76a8fd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb878034aa0 0x7fb8780c3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 10 monclient(hunting): picked mon.noname-a con 0x7fb878034aa0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:21.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb8780c3d00 0x7fb8780c0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 10 monclient(hunting): picked mon.noname-c con 0x7fb8780c3d00 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:21.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb878031f30 0x7fb87815ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 10 monclient(hunting): picked mon.noname-b con 0x7fb878031f30 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:21.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb878005a20 con 0x7fb878034aa0 2023-10-30T12:10:21.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8780c0a00 con 0x7fb878031f30 2023-10-30T12:10:21.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8780bf4b0 con 0x7fb8780c3d00 2023-10-30T12:10:21.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.244+0000 7fb8870df640 10 monclient(hunting): authenticate will time out at 1412.458582s 2023-10-30T12:10:21.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d0dc0640 1 -- 172.21.15.3:0/2349584406 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc031810 con 0x7f63c40392d0 2023-10-30T12:10:21.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d0dc0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d0dc0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:21.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d0dc0640 10 monclient: dump: 2023-10-30T12:10:21.348 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.349 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.349 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.349 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.349 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.349 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.350 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.350 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.350 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.350 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d0dc0640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d0dc0640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d0dc0640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d0dc0640 1 -- 172.21.15.3:0/2349584406 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc031980 con 0x7f63c40392d0 2023-10-30T12:10:21.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d4026640 5 monclient: authenticate success, global_id 4265 2023-10-30T12:10:21.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d4026640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:21.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d0dc0640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d0dc0640 1 -- 172.21.15.3:0/2349584406 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc031d20 con 0x7f63c40392d0 2023-10-30T12:10:21.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d0dc0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d0dc0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:21.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d0dc0640 10 monclient: dump: 2023-10-30T12:10:21.353 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.353 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.353 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.353 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.353 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.354 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.354 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.354 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.354 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.354 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d4026640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:21.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7f63d4026640 4 set_mon_vals no callback set 2023-10-30T12:10:21.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7d67fc640 1 -- 172.21.15.3:0/490902170 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d002eae0 con 0x7ff7d80b0990 2023-10-30T12:10:21.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:21.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7d67fc640 10 monclient: dump: 2023-10-30T12:10:21.357 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.357 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.357 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.357 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.358 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.358 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.358 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.358 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.358 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.359 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7d67fc640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7d67fc640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7d67fc640 1 -- 172.21.15.3:0/490902170 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7d002ec50 con 0x7ff7d80b0990 2023-10-30T12:10:21.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7ddf5a640 5 monclient: authenticate success, global_id 4315 2023-10-30T12:10:21.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7ddf5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:21.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7d67fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7ddf5a640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:21.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7ddf5a640 4 set_mon_vals no callback set 2023-10-30T12:10:21.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7d67fc640 1 -- 172.21.15.3:0/490902170 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d0009540 con 0x7ff7d80b0990 2023-10-30T12:10:21.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:21.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.245+0000 7ff7d67fc640 10 monclient: dump: 2023-10-30T12:10:21.361 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.361 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.362 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.362 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.362 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.362 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.362 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.363 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.363 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.363 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f63d4026640 10 monclient: shutdown 2023-10-30T12:10:21.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f63d4026640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:21.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f63d4026640 1 -- 172.21.15.3:0/2349584406 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c40392d0 msgr2=0x7f63c40c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f63d4026640 1 --2- 172.21.15.3:0/2349584406 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c40392d0 0x7f63c40c7dd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0097a0 tx=0x7f63bc02b8b0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7ff7ddf5a640 10 monclient: shutdown 2023-10-30T12:10:21.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7ff7ddf5a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:21.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7ff7ddf5a640 1 -- 172.21.15.3:0/490902170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7d80b0990 msgr2=0x7ff7d8071cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7ff7ddf5a640 1 --2- 172.21.15.3:0/490902170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7d80b0990 0x7ff7d8071cf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff7d000d050 tx=0x7ff7d002e140 comp rx=0 tx=0).stop 2023-10-30T12:10:21.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3ddd59a640 1 -- 172.21.15.3:0/390127575 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd4031750 con 0x7f3de0071180 2023-10-30T12:10:21.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7ff7ddf5a640 1 -- 172.21.15.3:0/490902170 shutdown_connections 2023-10-30T12:10:21.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7ff7ddf5a640 1 --2- 172.21.15.3:0/490902170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7d80b0990 0x7ff7d8071cf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7ff7ddf5a640 1 -- 172.21.15.3:0/490902170 >> 172.21.15.3:0/490902170 conn(0x7ff7d80b3c20 msgr2=0x7ff7d80b0010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:21.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3ddd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3ddd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:21.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3ddd59a640 10 monclient: dump: 2023-10-30T12:10:21.368 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.368 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.368 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.368 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.369 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.369 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.369 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.369 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.369 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.370 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3ddd59a640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3ddd59a640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3ddd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3ddd59a640 1 -- 172.21.15.3:0/390127575 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd40318c0 con 0x7f3de0071180 2023-10-30T12:10:21.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3ddd59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3ddd59a640 1 -- 172.21.15.3:0/390127575 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd4031ca0 con 0x7f3de0071180 2023-10-30T12:10:21.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3ddd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3ddd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:21.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3ddd59a640 10 monclient: dump: 2023-10-30T12:10:21.372 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.372 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.372 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.372 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.372 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.373 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.373 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.373 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.373 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.373 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3de4f05640 5 monclient: authenticate success, global_id 4259 2023-10-30T12:10:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3de4f05640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f63d4026640 1 -- 172.21.15.3:0/2349584406 shutdown_connections 2023-10-30T12:10:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f63d4026640 1 --2- 172.21.15.3:0/2349584406 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c40392d0 0x7f63c40c7dd0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f63d4026640 1 --2- 172.21.15.3:0/2349584406 >> v2:172.21.15.3:3300/0 conn(0x7f63c4038ef0 0x7f63c40c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f63d4026640 1 -- 172.21.15.3:0/2349584406 >> 172.21.15.3:0/2349584406 conn(0x7f63c4006180 msgr2=0x7f63c4006590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:21.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7f3de4f05640 4 set_mon_vals no callback set 2023-10-30T12:10:21.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.246+0000 7ff7ddf5a640 1 -- 172.21.15.3:0/490902170 shutdown_connections 2023-10-30T12:10:21.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.247+0000 7ff7ddf5a640 1 -- 172.21.15.3:0/490902170 wait complete. 2023-10-30T12:10:21.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.247+0000 7f63d4026640 1 -- 172.21.15.3:0/2349584406 shutdown_connections 2023-10-30T12:10:21.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.247+0000 7f63d4026640 1 -- 172.21.15.3:0/2349584406 wait complete. 2023-10-30T12:10:21.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.247+0000 7fbbf659c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbbf80a8e40 0x7fbbf813efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50470/0 (socket says 172.21.15.3:50470) 2023-10-30T12:10:21.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.247+0000 7fbbf659c640 1 -- 172.21.15.3:0/4025230614 learned_addr learned my addr 172.21.15.3:0/4025230614 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:21.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.247+0000 7f3de4f05640 10 monclient: shutdown 2023-10-30T12:10:21.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.247+0000 7f3de4f05640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:21.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.247+0000 7f3de4f05640 1 -- 172.21.15.3:0/390127575 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3de0071180 msgr2=0x7f3de0071560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.247+0000 7f3de4f05640 1 --2- 172.21.15.3:0/390127575 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3de0071180 0x7f3de0071560 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4009870 tx=0x7f3dd402b7f0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.247+0000 7ff7ddf5a640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.247+0000 7f3de4f05640 1 -- 172.21.15.3:0/390127575 shutdown_connections 2023-10-30T12:10:21.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.247+0000 7f3de4f05640 1 --2- 172.21.15.3:0/390127575 >> v2:172.21.15.3:3300/0 conn(0x7f3de0141f50 0x7f3de006f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7f3de4f05640 1 --2- 172.21.15.3:0/390127575 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3de0071180 0x7f3de0071560 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7f3de4f05640 1 -- 172.21.15.3:0/390127575 >> 172.21.15.3:0/390127575 conn(0x7f3de00b3a10 msgr2=0x7f3de00b3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:21.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7f63d4026640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.247+0000 7ff7ddf5a640 10 monclient: monmap: 2023-10-30T12:10:21.381 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:21.381 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:21.381 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:21.381 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:21.382 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.382 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:21.382 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:21.382 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:21.382 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:21.383 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7f63d4026640 10 monclient: monmap: 2023-10-30T12:10:21.383 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:21.384 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:21.384 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:21.384 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:21.384 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.384 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:21.385 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:21.385 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:21.385 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:21.385 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7f63d4026640 1 librados: starting msgr at 2023-10-30T12:10:21.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7f3de4f05640 1 -- 172.21.15.3:0/390127575 shutdown_connections 2023-10-30T12:10:21.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7f3de4f05640 1 -- 172.21.15.3:0/390127575 wait complete. 2023-10-30T12:10:21.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7ff7ddf5a640 1 librados: starting msgr at 2023-10-30T12:10:21.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7ff7ddf5a640 1 librados: starting objecter 2023-10-30T12:10:21.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7ff7ddf5a640 1 Processor -- start 2023-10-30T12:10:21.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7f63d4026640 1 librados: starting objecter 2023-10-30T12:10:21.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7ff7ddf5a640 1 -- start start 2023-10-30T12:10:21.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7ff7ddf5a640 1 librados: setting wanted keys 2023-10-30T12:10:21.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7ff7ddf5a640 1 librados: calling monclient init 2023-10-30T12:10:21.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7ff7ddf5a640 10 monclient: init 2023-10-30T12:10:21.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.248+0000 7f63d4026640 1 Processor -- start 2023-10-30T12:10:21.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7f63d4026640 1 -- start start 2023-10-30T12:10:21.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7f63d4026640 1 librados: setting wanted keys 2023-10-30T12:10:21.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7f63d4026640 1 librados: calling monclient init 2023-10-30T12:10:21.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7f63d4026640 10 monclient: init 2023-10-30T12:10:21.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7f3de4f05640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7f3de4f05640 10 monclient: monmap: 2023-10-30T12:10:21.391 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:21.391 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:21.391 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:21.391 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:21.392 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.392 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:21.392 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:21.392 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:21.392 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:21.393 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7ff7ddf5a640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7ff7ddf5a640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:10:21.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7fbbf6d9d640 1 --2- 172.21.15.3:0/4025230614 >> v2:172.21.15.92:3300/0 conn(0x7fbbf81423f0 0x7fbbf814c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7fbbf6d9d640 10 monclient(hunting): get_auth_request con 0x7fbbf81423f0 auth_method 0 2023-10-30T12:10:21.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7fbbf6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7fbbf6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7fbbf6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7f3de4f05640 1 librados: starting msgr at 2023-10-30T12:10:21.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7f3de4f05640 1 librados: starting objecter 2023-10-30T12:10:21.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.249+0000 7ff7ddf5a640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7d806eea0 0x7ff7d80acc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f3de4f05640 1 Processor -- start 2023-10-30T12:10:21.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7ff7ddf5a640 10 monclient(hunting): picked mon.noname-b con 0x7ff7d806eea0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:21.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f3de4f05640 1 -- start start 2023-10-30T12:10:21.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7ff7ddf5a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7d806f2c0 0x7ff7d81d6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:10:21.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c4038ef0 0x7f63c402df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f3de4f05640 1 librados: setting wanted keys 2023-10-30T12:10:21.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7ff7ddf5a640 10 monclient(hunting): picked mon.noname-c con 0x7ff7d806f2c0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:21.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f3de4f05640 1 librados: calling monclient init 2023-10-30T12:10:21.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f3de4f05640 10 monclient: init 2023-10-30T12:10:21.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7ff7ddf5a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d80b0990 0x7ff7d81d7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 10 monclient(hunting): picked mon.noname-a con 0x7f63c4038ef0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:21.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63c40c8310 0x7f63c40c1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 10 monclient(hunting): picked mon.noname-b con 0x7f63c40c8310 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:21.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c40c1ad0 0x7f63c4160300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7ff7ddf5a640 10 monclient(hunting): picked mon.noname-a con 0x7ff7d80b0990 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:21.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7ff7ddf5a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7ff7ddf5a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d8071eb0 con 0x7ff7d80b0990 2023-10-30T12:10:21.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 10 monclient(hunting): picked mon.noname-c con 0x7f63c40c1ad0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:21.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7ff7ddf5a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7ff7ddf5a640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d80a9e30 con 0x7ff7d806eea0 2023-10-30T12:10:21.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40399b0 con 0x7f63c4038ef0 2023-10-30T12:10:21.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7ff7ddf5a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7ff7ddf5a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d81d7940 con 0x7ff7d806f2c0 2023-10-30T12:10:21.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7ff7ddf5a640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40cbe50 con 0x7f63c40c8310 2023-10-30T12:10:21.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7ff7ddf5a640 10 monclient(hunting): authenticate will time out at 1412.464864s 2023-10-30T12:10:21.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c402f0e0 con 0x7f63c40c1ad0 2023-10-30T12:10:21.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f63d4026640 10 monclient(hunting): authenticate will time out at 1412.464921s 2023-10-30T12:10:21.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f9d8bfff640 1 --2- 172.21.15.3:0/2555943919 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d740053d0 0x7f9d7400ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f3de4f05640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.250+0000 7f3de4f05640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:10:21.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f3de4f05640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3de0071180 0x7f3de013cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f3de4f05640 10 monclient(hunting): picked mon.noname-a con 0x7f3de0071180 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:21.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f3de4f05640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3de0141b30 0x7f3de013f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f3de4f05640 10 monclient(hunting): picked mon.noname-b con 0x7f3de0141b30 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:21.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f3de4f05640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3de0141f50 0x7f3de00abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f3de4f05640 10 monclient(hunting): picked mon.noname-c con 0x7f3de0141f50 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:21.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f3de4f05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f3de4f05640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de006e5e0 con 0x7f3de0071180 2023-10-30T12:10:21.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f3de4f05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f3de4f05640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de013cb30 con 0x7f3de0141b30 2023-10-30T12:10:21.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f9d8bfff640 10 monclient(hunting): get_auth_request con 0x7f9d740053d0 auth_method 0 2023-10-30T12:10:21.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f9d8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f9d8bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f9d8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:21.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f3de4f05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f3de4f05640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de013ea20 con 0x7f3de0141f50 2023-10-30T12:10:21.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f3de4f05640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f3de4f05640 10 monclient(hunting): authenticate will time out at 1412.465457s 2023-10-30T12:10:21.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7fbbf5d9b640 1 --2- 172.21.15.3:0/4025230614 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf80b13c0 0x7fbbf8141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9b5ffb640 1 -- 172.21.15.3:0/2018086619 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8031740 con 0x7ff9b80b4350 2023-10-30T12:10:21.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9b5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:21.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9b5ffb640 10 monclient: dump: 2023-10-30T12:10:21.416 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.416 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.417 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.417 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.417 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.417 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.417 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.417 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.418 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.418 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9b5ffb640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9b5ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9b5ffb640 1 -- 172.21.15.3:0/2018086619 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a80318b0 con 0x7ff9b80b4350 2023-10-30T12:10:21.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9b5ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9b5ffb640 1 -- 172.21.15.3:0/2018086619 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8031c50 con 0x7ff9b80b4350 2023-10-30T12:10:21.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:21.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9b5ffb640 10 monclient: dump: 2023-10-30T12:10:21.420 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.420 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.420 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.420 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.421 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.421 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.421 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.421 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.421 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.421 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9bd42d640 5 monclient: authenticate success, global_id 4344 2023-10-30T12:10:21.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9bd42d640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:21.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7ff9bd42d640 4 set_mon_vals no callback set 2023-10-30T12:10:21.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7f9d8affd640 1 --2- 172.21.15.3:0/2555943919 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d740047c0 0x7f9d74004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.251+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.252+0000 7ff9bd42d640 10 monclient: shutdown 2023-10-30T12:10:21.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.252+0000 7ff9bd42d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:21.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.252+0000 7ff9bd42d640 1 -- 172.21.15.3:0/2018086619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b80b4350 msgr2=0x7ff9b80a83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.252+0000 7ff9bd42d640 1 --2- 172.21.15.3:0/2018086619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b80b4350 0x7ff9b80a83d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff9b80a9770 tx=0x7ff9a802b7e0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.252+0000 7fb8860dd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb878034aa0 0x7fb8780c3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.252+0000 7ff9bd42d640 1 -- 172.21.15.3:0/2018086619 shutdown_connections 2023-10-30T12:10:21.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.252+0000 7ff9bd42d640 1 --2- 172.21.15.3:0/2018086619 >> v2:172.21.15.92:3300/0 conn(0x7ff9b8144f50 0x7ff9b81453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.252+0000 7ff9bd42d640 1 --2- 172.21.15.3:0/2018086619 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9b813eb50 0x7ff9b813ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.252+0000 7ff9bd42d640 1 --2- 172.21.15.3:0/2018086619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b80b4350 0x7ff9b80a83d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.252+0000 7ff9bd42d640 1 -- 172.21.15.3:0/2018086619 >> 172.21.15.3:0/2018086619 conn(0x7ff9b8065ea0 msgr2=0x7ff9b80662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:21.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.252+0000 7f9d8b7fe640 1 --2- 172.21.15.3:0/2555943919 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d740039c0 0x7f9d74003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.252+0000 7ff9bd42d640 1 -- 172.21.15.3:0/2018086619 shutdown_connections 2023-10-30T12:10:21.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.252+0000 7ff9bd42d640 1 -- 172.21.15.3:0/2018086619 wait complete. 2023-10-30T12:10:21.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.253+0000 7fb8858dc640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb8780c3d00 0x7fb8780c0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.253+0000 7ff9bd42d640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.253+0000 7ff9bd42d640 10 monclient: monmap: 2023-10-30T12:10:21.428 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:21.428 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:21.428 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:21.429 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:21.429 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.429 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:21.429 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:21.429 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:21.430 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:21.430 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.253+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_done global_id 4330 payload 274 2023-10-30T12:10:21.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.253+0000 7fbbf6d9d640 1 -- 172.21.15.3:0/4025230614 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf80b13c0 msgr2=0x7fbbf8141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:21.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.253+0000 7fbbf6d9d640 1 --2- 172.21.15.3:0/4025230614 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf80b13c0 0x7fbbf8141eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.253+0000 7fbbf6d9d640 1 -- 172.21.15.3:0/4025230614 >> v1:172.21.15.3:6789/0 conn(0x7fbbf80a8e40 legacy=0x7fbbf813efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.253+0000 7fbbf6d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.253+0000 7fbbf6d9d640 1 monclient: found mon.noname-c 2023-10-30T12:10:21.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.253+0000 7fbbf6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.253+0000 7fbbf6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.253+0000 7fbbf6d9d640 1 -- 172.21.15.3:0/4025230614 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf80a8920 con 0x7fbbf81423f0 2023-10-30T12:10:21.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.253+0000 7ff9bd42d640 1 librados: starting msgr at 2023-10-30T12:10:21.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.253+0000 7ff9bd42d640 1 librados: starting objecter 2023-10-30T12:10:21.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7fb8858dc640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb8780c3d00 0x7fb8780c0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:60882/0 (socket says 172.21.15.3:60882) 2023-10-30T12:10:21.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7fb8858dc640 1 -- 172.21.15.3:0/164245000 learned_addr learned my addr 172.21.15.3:0/164245000 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:21.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7fb8858dc640 10 monclient(hunting): get_auth_request con 0x7fb8780c3d00 auth_method 0 2023-10-30T12:10:21.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7fb8858dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7fb8858dc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7fb8858dc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7fb8868de640 1 --2- 172.21.15.3:0/164245000 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb878031f30 0x7fb87815ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7ff9bd42d640 1 Processor -- start 2023-10-30T12:10:21.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7f9d8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7f9d8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7f9d8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:10:21.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7ff9bd42d640 1 -- start start 2023-10-30T12:10:21.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7ff9bd42d640 1 librados: setting wanted keys 2023-10-30T12:10:21.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7ff9bd42d640 1 librados: calling monclient init 2023-10-30T12:10:21.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7ff9bd42d640 10 monclient: init 2023-10-30T12:10:21.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7f9d8b7fe640 10 monclient(hunting): get_auth_request con 0x7f9d740039c0 auth_method 0 2023-10-30T12:10:21.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7f9d8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7f9d8b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7f9d8b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:21.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7f9d8affd640 10 monclient(hunting): get_auth_request con 0x7f9d740047c0 auth_method 0 2023-10-30T12:10:21.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7f9d8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7f9d8affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7f9d8affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:21.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7fb8860dd640 10 monclient(hunting): get_auth_request con 0x7fb878034aa0 auth_method 0 2023-10-30T12:10:21.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7fb8860dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7fb8860dd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7ff9bd42d640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7ff9bd42d640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:10:21.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7fb8860dd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7fb8868de640 10 monclient(hunting): get_auth_request con 0x7fb878031f30 auth_method 0 2023-10-30T12:10:21.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7fb8868de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7fb8868de640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7fb8868de640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.254+0000 7ff9bd42d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9b80b4350 0x7ff9b80b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7ff9bd42d640 10 monclient(hunting): picked mon.noname-c con 0x7ff9b80b4350 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:21.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7ff9bd42d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b813eb50 0x7ff9b80b5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7ff9bd42d640 10 monclient(hunting): picked mon.noname-a con 0x7ff9b813eb50 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:21.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7ff9bd42d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9b8144f50 0x7ff9b80b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7ff9bd42d640 10 monclient(hunting): picked mon.noname-b con 0x7ff9b8144f50 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:21.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7ff9bd42d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7ff9bd42d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b81453c0 con 0x7ff9b813eb50 2023-10-30T12:10:21.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7ff9bd42d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7ff9bd42d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b80b0920 con 0x7ff9b8144f50 2023-10-30T12:10:21.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7ff9bd42d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7ff9bd42d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b80b6250 con 0x7ff9b80b4350 2023-10-30T12:10:21.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7ff9bd42d640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.255+0000 7ff9bd42d640 10 monclient(hunting): authenticate will time out at 1412.469210s 2023-10-30T12:10:21.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7ff7d77fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7d806eea0 0x7ff7d80acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9bfeffd640 1 -- 172.21.15.3:0/2589113220 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec031520 con 0x7f9bf8036280 2023-10-30T12:10:21.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9bfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9bfeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:21.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9bfeffd640 10 monclient: dump: 2023-10-30T12:10:21.446 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.446 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.446 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.447 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.447 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.447 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.447 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.447 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.447 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.448 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9bfeffd640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9bfeffd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9bfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9bfeffd640 1 -- 172.21.15.3:0/2589113220 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bec02f050 con 0x7f9bf8036280 2023-10-30T12:10:21.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9bfeffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9bfeffd640 1 -- 172.21.15.3:0/2589113220 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec0318b0 con 0x7f9bf8036280 2023-10-30T12:10:21.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9bfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9bfeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:21.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9bfeffd640 10 monclient: dump: 2023-10-30T12:10:21.450 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.450 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.450 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.450 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.450 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.451 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.451 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.451 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.451 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.451 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9c067b0640 5 monclient: authenticate success, global_id 4324 2023-10-30T12:10:21.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9c067b0640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:21.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9c067b0640 4 set_mon_vals no callback set 2023-10-30T12:10:21.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7fb8858dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7fb8858dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7fb8858dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9d8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9d8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9d8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:10:21.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9c067b0640 10 monclient: shutdown 2023-10-30T12:10:21.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9c067b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:21.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9c067b0640 1 -- 172.21.15.3:0/2589113220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf8036280 msgr2=0x7f9bf80d68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9c067b0640 1 --2- 172.21.15.3:0/2589113220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf8036280 0x7f9bf80d68e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9bf80cd130 tx=0x7f9bec02b7c0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9c067b0640 1 -- 172.21.15.3:0/2589113220 shutdown_connections 2023-10-30T12:10:21.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9c067b0640 1 --2- 172.21.15.3:0/2589113220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf8036280 0x7f9bf80d68e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9c067b0640 1 --2- 172.21.15.3:0/2589113220 >> v2:172.21.15.3:3300/0 conn(0x7f9bf80070a0 0x7f9bf80338a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.256+0000 7f9c067b0640 1 -- 172.21.15.3:0/2589113220 >> 172.21.15.3:0/2589113220 conn(0x7f9bf8006040 msgr2=0x7f9bf8006450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:21.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9c067b0640 1 -- 172.21.15.3:0/2589113220 shutdown_connections 2023-10-30T12:10:21.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9c067b0640 1 -- 172.21.15.3:0/2589113220 wait complete. 2023-10-30T12:10:21.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:10:21.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8bfff640 10 monclient(hunting): handle_auth_done global_id 4327 payload 293 2023-10-30T12:10:21.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8bfff640 1 -- 172.21.15.3:0/2555943919 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d740039c0 msgr2=0x7f9d74003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:21.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8bfff640 1 --2- 172.21.15.3:0/2555943919 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d740039c0 0x7f9d74003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8bfff640 1 -- 172.21.15.3:0/2555943919 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d740047c0 msgr2=0x7f9d74004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:21.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8bfff640 1 --2- 172.21.15.3:0/2555943919 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d740047c0 0x7f9d74004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8bfff640 1 monclient: found mon.b 2023-10-30T12:10:21.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8bfff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8bfff640 1 -- 172.21.15.3:0/2555943919 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c001510 con 0x7f9d740053d0 2023-10-30T12:10:21.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8bfff640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9d8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7fb8860dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7fb8860dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7fb8860dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9c067b0640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f9c067b0640 10 monclient: monmap: 2023-10-30T12:10:21.461 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:21.461 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:21.461 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:21.462 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:21.462 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.462 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:21.462 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:21.462 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:21.463 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:21.463 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7f9c067b0640 1 librados: starting msgr at 2023-10-30T12:10:21.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7f9c067b0640 1 librados: starting objecter 2023-10-30T12:10:21.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.257+0000 7f63d1dc2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c4038ef0 0x7f63c402df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7f9c067b0640 1 Processor -- start 2023-10-30T12:10:21.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7fb8868de640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7fb8868de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7fb8868de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7f9d8bfff640 1 -- 172.21.15.3:0/2555943919 reap_dead start 2023-10-30T12:10:21.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7f9c067b0640 1 -- start start 2023-10-30T12:10:21.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7f9c067b0640 1 librados: setting wanted keys 2023-10-30T12:10:21.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7f9c067b0640 1 librados: calling monclient init 2023-10-30T12:10:21.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7f9c067b0640 10 monclient: init 2023-10-30T12:10:21.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7ff7d77fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7d806eea0 0x7ff7d80acc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56382/0 (socket says 172.21.15.3:56382) 2023-10-30T12:10:21.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7ff7d77fe640 1 -- 172.21.15.3:0/1611936587 learned_addr learned my addr 172.21.15.3:0/1611936587 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:21.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7ff7d77fe640 10 monclient(hunting): get_auth_request con 0x7ff7d806eea0 auth_method 0 2023-10-30T12:10:21.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7ff7d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7ff7d77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.258+0000 7ff7d77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:21.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf80070a0 0x7f9bf80cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 10 monclient(hunting): picked mon.noname-b con 0x7f9bf80070a0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:21.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf8036280 0x7f9bf80ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 10 monclient(hunting): picked mon.noname-a con 0x7f9bf8036280 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:21.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9bf80cc960 0x7f9bf80ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 10 monclient(hunting): picked mon.noname-c con 0x7f9bf80cc960 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:21.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf8000b90 con 0x7f9bf8036280 2023-10-30T12:10:21.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf80ccde0 con 0x7f9bf80070a0 2023-10-30T12:10:21.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf8164bd0 con 0x7f9bf80cc960 2023-10-30T12:10:21.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.259+0000 7f9c067b0640 10 monclient(hunting): authenticate will time out at 1412.473602s 2023-10-30T12:10:21.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7f63d1dc2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c4038ef0 0x7f63c402df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42392/0 (socket says 172.21.15.3:42392) 2023-10-30T12:10:21.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7f63d1dc2640 1 -- 172.21.15.3:0/1163036885 learned_addr learned my addr 172.21.15.3:0/1163036885 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:21.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7f63d1dc2640 10 monclient(hunting): get_auth_request con 0x7f63c4038ef0 auth_method 0 2023-10-30T12:10:21.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7f63d1dc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7f63d1dc2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7f63d1dc2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7fb8868de640 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2023-10-30T12:10:21.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7fb8868de640 1 -- 172.21.15.3:0/164245000 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb8780c3d00 msgr2=0x7fb8780c0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7fb8868de640 1 --2- 172.21.15.3:0/164245000 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb8780c3d00 0x7fb8780c0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7fb8868de640 1 -- 172.21.15.3:0/164245000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb878034aa0 msgr2=0x7fb8780c3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7fb8868de640 1 --2- 172.21.15.3:0/164245000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb878034aa0 0x7fb8780c3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7fb8868de640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7fb8868de640 1 monclient: found mon.noname-b 2023-10-30T12:10:21.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7fb8868de640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7fb8868de640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.261+0000 7fb8868de640 1 -- 172.21.15.3:0/164245000 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8780c14e0 con 0x7fb878031f30 2023-10-30T12:10:21.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.262+0000 7f3dddd9b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3de0141b30 0x7f3de013f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.263+0000 7ff7d7fff640 1 --2- 172.21.15.3:0/1611936587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d80b0990 0x7ff7d81d7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.263+0000 7f63d1dc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.263+0000 7f63d1dc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.263+0000 7f63d1dc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.264+0000 7f63d15c1640 1 --2- 172.21.15.3:0/1163036885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63c40c8310 0x7f63c40c1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.264+0000 7f3dddd9b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3de0141b30 0x7f3de013f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56400/0 (socket says 172.21.15.3:56400) 2023-10-30T12:10:21.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.264+0000 7f3dddd9b640 1 -- 172.21.15.3:0/2538696770 learned_addr learned my addr 172.21.15.3:0/2538696770 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:21.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.264+0000 7f3dddd9b640 10 monclient(hunting): get_auth_request con 0x7f3de0141b30 auth_method 0 2023-10-30T12:10:21.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.264+0000 7f3dddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.264+0000 7f3dddd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.264+0000 7f3dddd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.264+0000 7f63d15c1640 10 monclient(hunting): get_auth_request con 0x7f63c40c8310 auth_method 0 2023-10-30T12:10:21.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.264+0000 7f63d15c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.264+0000 7f63d15c1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.264+0000 7f63d15c1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.265+0000 7f3dddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.265+0000 7f3dddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.265+0000 7f3dddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.266+0000 7ff7d7fff640 10 monclient(hunting): get_auth_request con 0x7ff7d80b0990 auth_method 0 2023-10-30T12:10:21.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.266+0000 7ff7d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.266+0000 7ff7d7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.266+0000 7ff7d7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.266+0000 7ff7d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.266+0000 7ff7d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.266+0000 7ff7d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.266+0000 7ff9b77fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9b8144f50 0x7ff9b80b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.267+0000 7f63d15c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.267+0000 7f63d15c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.267+0000 7f63d15c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.267+0000 7ff7d6ffd640 1 --2- 172.21.15.3:0/1611936587 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7d806f2c0 0x7ff7d81d6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.267+0000 7f63d25c3640 1 --2- 172.21.15.3:0/1163036885 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c40c1ad0 0x7f63c4160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.267+0000 7ff9b77fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9b8144f50 0x7ff9b80b7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56404/0 (socket says 172.21.15.3:56404) 2023-10-30T12:10:21.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.267+0000 7ff9b77fe640 1 -- 172.21.15.3:0/3554776335 learned_addr learned my addr 172.21.15.3:0/3554776335 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:21.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.267+0000 7ff9b77fe640 10 monclient(hunting): get_auth_request con 0x7ff9b8144f50 auth_method 0 2023-10-30T12:10:21.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.267+0000 7ff9b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.267+0000 7ff9b77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.267+0000 7ff9b77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.268+0000 7ff7d6ffd640 10 monclient(hunting): get_auth_request con 0x7ff7d806f2c0 auth_method 0 2023-10-30T12:10:21.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.268+0000 7ff7d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.268+0000 7ff7d6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.268+0000 7ff7d6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.268+0000 7ff7d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.268+0000 7ff7d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.268+0000 7ff7d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.268+0000 7f63d25c3640 10 monclient(hunting): get_auth_request con 0x7f63c40c1ad0 auth_method 0 2023-10-30T12:10:21.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.268+0000 7f63d25c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.268+0000 7f63d25c3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.268+0000 7f63d25c3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.269+0000 7ff7d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.269+0000 7ff7d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.269+0000 7ff7d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.270+0000 7f63d25c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.270+0000 7f63d25c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.270+0000 7f63d25c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.270+0000 7f63d15c1640 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2023-10-30T12:10:21.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.270+0000 7f63d15c1640 1 -- 172.21.15.3:0/1163036885 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c40c1ad0 msgr2=0x7f63c4160300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.270+0000 7f63d15c1640 1 --2- 172.21.15.3:0/1163036885 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c40c1ad0 0x7f63c4160300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.270+0000 7f63d15c1640 1 -- 172.21.15.3:0/1163036885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c4038ef0 msgr2=0x7f63c402df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.270+0000 7f63d15c1640 1 --2- 172.21.15.3:0/1163036885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c4038ef0 0x7f63c402df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.270+0000 7f63d15c1640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.270+0000 7f63d15c1640 1 monclient: found mon.noname-b 2023-10-30T12:10:21.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.270+0000 7f63d15c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.270+0000 7f63d15c1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.270+0000 7f63d15c1640 1 -- 172.21.15.3:0/1163036885 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c40c1f40 con 0x7f63c40c8310 2023-10-30T12:10:21.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.272+0000 7ff7d7fff640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2023-10-30T12:10:21.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.272+0000 7ff7d7fff640 1 -- 172.21.15.3:0/1611936587 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7d806f2c0 msgr2=0x7ff7d81d6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.272+0000 7ff7d7fff640 1 --2- 172.21.15.3:0/1611936587 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7d806f2c0 0x7ff7d81d6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.272+0000 7ff7d7fff640 1 -- 172.21.15.3:0/1611936587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7d806eea0 msgr2=0x7ff7d80acc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.272+0000 7ff7d7fff640 1 --2- 172.21.15.3:0/1611936587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7d806eea0 0x7ff7d80acc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.272+0000 7ff7d7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.272+0000 7ff7d7fff640 1 monclient: found mon.noname-a 2023-10-30T12:10:21.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.272+0000 7ff7d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.272+0000 7ff7d7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:21.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.272+0000 7ff7d7fff640 1 -- 172.21.15.3:0/1611936587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d80a9630 con 0x7ff7d80b0990 2023-10-30T12:10:21.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.274+0000 7f3dddd9b640 10 monclient(hunting): handle_auth_done global_id 4336 payload 824 2023-10-30T12:10:21.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.274+0000 7f3dddd9b640 1 -- 172.21.15.3:0/2538696770 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3de0141f50 msgr2=0x7f3de00abb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.274+0000 7f3dddd9b640 1 --2- 172.21.15.3:0/2538696770 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3de0141f50 0x7f3de00abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.274+0000 7f3dddd9b640 1 -- 172.21.15.3:0/2538696770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3de0071180 msgr2=0x7f3de013cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.274+0000 7f3dddd9b640 1 --2- 172.21.15.3:0/2538696770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3de0071180 0x7f3de013cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.274+0000 7f3dddd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.274+0000 7f3dddd9b640 1 monclient: found mon.noname-b 2023-10-30T12:10:21.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.274+0000 7f3dddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.274+0000 7f3dddd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.274+0000 7f3dddd9b640 1 -- 172.21.15.3:0/2538696770 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de00a83d0 con 0x7f3de0141b30 2023-10-30T12:10:21.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.277+0000 7ff9b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.277+0000 7ff9b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.278+0000 7ff9b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.278+0000 7fb8868de640 1 --2- 172.21.15.3:0/164245000 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb878031f30 0x7fb87815ade0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb880067f90 tx=0x7fb880067340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.278+0000 7fb876ffd640 10 client.?.objecter ms_handle_connect 0x7fb878031f30 2023-10-30T12:10:21.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.278+0000 7fb876ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:21.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.279+0000 7ff7d7fff640 1 --2- 172.21.15.3:0/1611936587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d80b0990 0x7ff7d81d7400 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800c3f0 tx=0x7ff7c800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.279+0000 7ff7bffff640 10 client.?.objecter ms_handle_connect 0x7ff7d80b0990 2023-10-30T12:10:21.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.279+0000 7ff7bffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:21.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.279+0000 7ff9b6ffd640 1 --2- 172.21.15.3:0/3554776335 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9b80b4350 0x7ff9b80b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.280+0000 7f9c04d4d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9bf80cc960 0x7f9bf80ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.280+0000 7ff9b77fe640 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2023-10-30T12:10:21.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.280+0000 7ff9b77fe640 1 -- 172.21.15.3:0/3554776335 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9b80b4350 msgr2=0x7ff9b80b53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.280+0000 7ff9b77fe640 1 --2- 172.21.15.3:0/3554776335 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9b80b4350 0x7ff9b80b53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.281+0000 7ff9b77fe640 1 -- 172.21.15.3:0/3554776335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b813eb50 msgr2=0x7ff9b80b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.281+0000 7ff9b77fe640 1 --2- 172.21.15.3:0/3554776335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b813eb50 0x7ff9b80b5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.281+0000 7ff9b77fe640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.281+0000 7ff9b77fe640 1 monclient: found mon.noname-b 2023-10-30T12:10:21.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.281+0000 7ff9b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.281+0000 7ff9b77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.281+0000 7ff9b77fe640 1 -- 172.21.15.3:0/3554776335 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b8137590 con 0x7ff9b8144f50 2023-10-30T12:10:21.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.281+0000 7f3dddd9b640 1 --2- 172.21.15.3:0/2538696770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3de0141b30 0x7f3de013f2e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000c430 tx=0x7f3dd000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.281+0000 7f3dceffd640 10 client.?.objecter ms_handle_connect 0x7f3de0141b30 2023-10-30T12:10:21.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.281+0000 7f3dceffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:21.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.283+0000 7f9c04d4d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9bf80cc960 0x7f9bf80ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:60936/0 (socket says 172.21.15.3:60936) 2023-10-30T12:10:21.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.283+0000 7f9c04d4d640 1 -- 172.21.15.3:0/614844551 learned_addr learned my addr 172.21.15.3:0/614844551 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:21.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.283+0000 7f9c04d4d640 10 monclient(hunting): get_auth_request con 0x7f9bf80cc960 auth_method 0 2023-10-30T12:10:21.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.283+0000 7f9c04d4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.283+0000 7f9c04d4d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.283+0000 7f9c04d4d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7bffff640 1 -- 172.21.15.3:0/1611936587 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8005ce0 con 0x7ff7d80b0990 2023-10-30T12:10:21.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:21.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7bffff640 10 monclient: dump: 2023-10-30T12:10:21.508 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.508 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.509 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.509 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.509 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.509 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.509 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.510 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.510 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.510 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7bffff640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7bffff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7bffff640 1 -- 172.21.15.3:0/1611936587 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c800e040 con 0x7ff7d80b0990 2023-10-30T12:10:21.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7bffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7d5ffb640 4 set_mon_vals no callback set 2023-10-30T12:10:21.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7f9c04d4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7f9c04d4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7f9c04d4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7ddf5a640 5 monclient: authenticate success, global_id 4356 2023-10-30T12:10:21.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7bffff640 1 -- 172.21.15.3:0/1611936587 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c80072f0 con 0x7ff7d80b0990 2023-10-30T12:10:21.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7ddf5a640 10 monclient: _renew_subs 2023-10-30T12:10:21.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7ddf5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:21.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7ddf5a640 1 -- 172.21.15.3:0/1611936587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d80ad160 con 0x7ff7d80b0990 2023-10-30T12:10:21.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:21.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7bffff640 10 monclient: dump: 2023-10-30T12:10:21.515 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.515 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.515 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.515 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.515 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.516 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.516 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.516 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.516 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.516 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7ddf5a640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:21.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7ddf5a640 10 monclient: _renew_subs 2023-10-30T12:10:21.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7ddf5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:21.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7ddf5a640 1 -- 172.21.15.3:0/1611936587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d8136910 con 0x7ff7d80b0990 2023-10-30T12:10:21.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.284+0000 7ff7ddf5a640 1 librados: init done 2023-10-30T12:10:21.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.285+0000 7ff7ddf5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.286+0000 7ff7ddf5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.286+0000 7ff7ddf5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:21.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.286+0000 7ff7ddf5a640 1 -- 172.21.15.3:0/1611936587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d804e9b0 con 0x7ff7d80b0990 2023-10-30T12:10:21.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.288+0000 7f9c04d4d640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2023-10-30T12:10:21.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.288+0000 7f9c04d4d640 1 -- 172.21.15.3:0/614844551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf80070a0 msgr2=0x7f9bf80cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.288+0000 7f9c04d4d640 1 --2- 172.21.15.3:0/614844551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf80070a0 0x7f9bf80cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.288+0000 7f9c04d4d640 1 -- 172.21.15.3:0/614844551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf8036280 msgr2=0x7f9bf80ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.288+0000 7f9c04d4d640 1 --2- 172.21.15.3:0/614844551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf8036280 0x7f9bf80ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.288+0000 7f9c04d4d640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.288+0000 7f9c04d4d640 1 monclient: found mon.noname-c 2023-10-30T12:10:21.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.288+0000 7f9c04d4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.288+0000 7f9c04d4d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.3:3301/0 2023-10-30T12:10:21.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.288+0000 7f9c04d4d640 1 -- 172.21.15.3:0/614844551 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf8164da0 con 0x7f9bf80cc960 2023-10-30T12:10:21.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.289+0000 7f9d8bfff640 1 --2- 172.21.15.3:0/2555943919 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d740053d0 0x7f9d7400ba30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9d80001650 tx=0x7f9d740057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18c80fe589f763e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.291+0000 7f9c04d4d640 1 --2- 172.21.15.3:0/614844551 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9bf80cc960 0x7f9bf80ca270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9bec02bdd0 tx=0x7f9bec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.291+0000 7f9bfcff9640 10 client.?.objecter ms_handle_connect 0x7f9bf80cc960 2023-10-30T12:10:21.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.291+0000 7f9bfcff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:21.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3dceffd640 1 -- 172.21.15.3:0/2538696770 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0011790 con 0x7f3de0141b30 2023-10-30T12:10:21.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3dceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3dceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:21.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3dceffd640 10 monclient: dump: 2023-10-30T12:10:21.524 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.524 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.525 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.525 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.525 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.525 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.525 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.526 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.526 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.526 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3dceffd640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3dceffd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3dceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3dceffd640 1 -- 172.21.15.3:0/2538696770 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd000e040 con 0x7f3de0141b30 2023-10-30T12:10:21.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3dceffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3dceffd640 1 -- 172.21.15.3:0/2538696770 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0011b40 con 0x7f3de0141b30 2023-10-30T12:10:21.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3dceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3dceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:21.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3dceffd640 10 monclient: dump: 2023-10-30T12:10:21.528 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.528 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.528 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.529 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.529 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.529 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.529 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.529 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.529 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.530 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3ddcd99640 4 set_mon_vals no callback set 2023-10-30T12:10:21.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3de4f05640 5 monclient: authenticate success, global_id 4336 2023-10-30T12:10:21.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3de4f05640 10 monclient: _renew_subs 2023-10-30T12:10:21.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3de4f05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.292+0000 7f3de4f05640 1 -- 172.21.15.3:0/2538696770 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de0141020 con 0x7f3de0141b30 2023-10-30T12:10:21.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.293+0000 7f3de4f05640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:21.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.293+0000 7f3de4f05640 10 monclient: _renew_subs 2023-10-30T12:10:21.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.293+0000 7f3de4f05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.293+0000 7f3de4f05640 1 -- 172.21.15.3:0/2538696770 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de013c880 con 0x7f3de0141b30 2023-10-30T12:10:21.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.293+0000 7f3de4f05640 1 librados: init done 2023-10-30T12:10:21.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.294+0000 7f3de4f05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.294+0000 7f3de4f05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.294+0000 7f3de4f05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.294+0000 7f3de4f05640 1 -- 172.21.15.3:0/2538696770 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de013d570 con 0x7f3de0141b30 2023-10-30T12:10:21.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.295+0000 7f63d15c1640 1 --2- 172.21.15.3:0/1163036885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63c40c8310 0x7f63c40c1590 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f63c40378d0 tx=0x7f63bc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.295+0000 7f63c27fc640 10 client.?.objecter ms_handle_connect 0x7f63c40c8310 2023-10-30T12:10:21.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.295+0000 7f63c27fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:21.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.295+0000 7f9bfcff9640 1 -- 172.21.15.3:0/614844551 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec0338e0 con 0x7f9bf80cc960 2023-10-30T12:10:21.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.295+0000 7f9bfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.295+0000 7f9bfcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:21.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9bfcff9640 10 monclient: dump: 2023-10-30T12:10:21.536 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.536 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.536 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.536 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.537 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.537 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.537 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.537 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.537 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.538 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9bfcff9640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9bfcff9640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9bfcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9bfcff9640 1 -- 172.21.15.3:0/614844551 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bec033a70 con 0x7f9bf80cc960 2023-10-30T12:10:21.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9c067b0640 5 monclient: authenticate success, global_id 4277 2023-10-30T12:10:21.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9c067b0640 10 monclient: _renew_subs 2023-10-30T12:10:21.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9c067b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:21.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9c067b0640 1 -- 172.21.15.3:0/614844551 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf8164f40 con 0x7f9bf80cc960 2023-10-30T12:10:21.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9bfcff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9bfcff9640 1 -- 172.21.15.3:0/614844551 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec033e30 con 0x7f9bf80cc960 2023-10-30T12:10:21.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9bfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9bfe7fc640 4 set_mon_vals no callback set 2023-10-30T12:10:21.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9c067b0640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:21.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9bfcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:21.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9bfcff9640 10 monclient: dump: 2023-10-30T12:10:21.541 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.541 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.541 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.541 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.541 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.541 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.542 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.542 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.542 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.542 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9c067b0640 10 monclient: _renew_subs 2023-10-30T12:10:21.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9c067b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:21.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9c067b0640 1 -- 172.21.15.3:0/614844551 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf80c1fc0 con 0x7f9bf80cc960 2023-10-30T12:10:21.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.296+0000 7f9c067b0640 1 librados: init done 2023-10-30T12:10:21.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.297+0000 7f9c067b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.297+0000 7f9c067b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.297+0000 7f9c067b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:21.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.297+0000 7f9c067b0640 1 -- 172.21.15.3:0/614844551 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf80ccf30 con 0x7f9bf80cc960 2023-10-30T12:10:21.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.298+0000 7ff9b77fe640 1 --2- 172.21.15.3:0/3554776335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9b8144f50 0x7ff9b80b7bf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000cc60 tx=0x7ff9b000a5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.298+0000 7ff99f7fe640 10 client.?.objecter ms_handle_connect 0x7ff9b8144f50 2023-10-30T12:10:21.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.298+0000 7ff99f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:21.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d8a7fc640 1 -- 172.21.15.3:0/2555943919 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8002e070 con 0x7f9d740053d0 2023-10-30T12:10:21.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d8a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:21.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d8a7fc640 10 monclient: dump: 2023-10-30T12:10:21.548 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.548 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.548 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.549 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.549 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.549 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.549 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.549 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.550 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.550 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d91a94640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:21.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d8a7fc640 1 -- 172.21.15.3:0/2555943919 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d80005270 con 0x7f9d740053d0 2023-10-30T12:10:21.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d8a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d91a94640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:21.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d91a94640 4 set_mon_vals no callback set 2023-10-30T12:10:21.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d8a7fc640 1 -- 172.21.15.3:0/2555943919 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80003610 con 0x7f9d740053d0 2023-10-30T12:10:21.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d8a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:21.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d8a7fc640 10 monclient: dump: 2023-10-30T12:10:21.552 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.552 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.552 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.552 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.552 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.553 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.553 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.553 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.553 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.553 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d91a94640 10 monclient: shutdown 2023-10-30T12:10:21.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d91a94640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:21.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d91a94640 1 -- 172.21.15.3:0/2555943919 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d740053d0 msgr2=0x7f9d7400ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d91a94640 1 --2- 172.21.15.3:0/2555943919 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d740053d0 0x7f9d7400ba30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9d80001650 tx=0x7f9d740057f0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d91a94640 1 -- 172.21.15.3:0/2555943919 shutdown_connections 2023-10-30T12:10:21.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d91a94640 1 --2- 172.21.15.3:0/2555943919 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d740053d0 0x7f9d7400ba30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d91a94640 1 -- 172.21.15.3:0/2555943919 >> 172.21.15.3:0/2555943919 conn(0x7f9d8c065ea0 msgr2=0x7f9d8c0662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:21.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.300+0000 7f9d91a94640 1 -- 172.21.15.3:0/2555943919 shutdown_connections 2023-10-30T12:10:21.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.301+0000 7f9d91a94640 1 -- 172.21.15.3:0/2555943919 wait complete. 2023-10-30T12:10:21.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.301+0000 7f9d91a94640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.301+0000 7f9d91a94640 10 monclient: monmap: 2023-10-30T12:10:21.556 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:21.556 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:21.556 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:21.556 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:21.556 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.556 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:21.557 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:21.557 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:21.557 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:21.557 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.302+0000 7f9d91a94640 1 librados: starting msgr at 2023-10-30T12:10:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.302+0000 7f9d91a94640 1 librados: starting objecter 2023-10-30T12:10:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.302+0000 7f9d91a94640 1 Processor -- start 2023-10-30T12:10:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.302+0000 7f9d91a94640 1 -- start start 2023-10-30T12:10:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.302+0000 7f9d91a94640 1 librados: setting wanted keys 2023-10-30T12:10:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.302+0000 7f9d91a94640 1 librados: calling monclient init 2023-10-30T12:10:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.302+0000 7f9d91a94640 10 monclient: init 2023-10-30T12:10:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.302+0000 7f9d91a94640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.302+0000 7f9d91a94640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.302+0000 7f9d91a94640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d740053d0 0x7f9d8c1cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.302+0000 7f9d91a94640 10 monclient(hunting): picked mon.noname-b con 0x7f9d740053d0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.302+0000 7f9d91a94640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d8c0acc00 0x7f9d8c0ad040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.303+0000 7f9d91a94640 10 monclient(hunting): picked mon.noname-a con 0x7f9d8c0acc00 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.303+0000 7f9d91a94640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d8c141df0 0x7f9d8c142270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.303+0000 7f9d91a94640 10 monclient(hunting): picked mon.noname-c con 0x7f9d8c141df0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.303+0000 7f9d91a94640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.303+0000 7f9d91a94640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c066280 con 0x7f9d8c0acc00 2023-10-30T12:10:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.303+0000 7f9d91a94640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.303+0000 7f9d91a94640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0ad630 con 0x7f9d740053d0 2023-10-30T12:10:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.303+0000 7f9d91a94640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.303+0000 7f9d91a94640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0ac950 con 0x7f9d8c141df0 2023-10-30T12:10:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.303+0000 7f9d91a94640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.303+0000 7f9d91a94640 10 monclient(hunting): authenticate will time out at 1412.517177s 2023-10-30T12:10:21.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.320+0000 7f9d8bfff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d8c141df0 0x7f9d8c142270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb876ffd640 1 -- 172.21.15.3:0/164245000 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8800723d0 con 0x7fb878031f30 2023-10-30T12:10:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb876ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb876ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:21.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb876ffd640 10 monclient: dump: 2023-10-30T12:10:21.564 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.564 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.564 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.564 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.564 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.565 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.565 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.565 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.565 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.565 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb876ffd640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb876ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb876ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb8870df640 5 monclient: authenticate success, global_id 4333 2023-10-30T12:10:21.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb8870df640 10 monclient: _renew_subs 2023-10-30T12:10:21.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb8870df640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb8870df640 1 -- 172.21.15.3:0/164245000 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87815b320 con 0x7fb878031f30 2023-10-30T12:10:21.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb8870df640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:21.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb8870df640 10 monclient: _renew_subs 2023-10-30T12:10:21.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb8870df640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb8870df640 1 -- 172.21.15.3:0/164245000 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8780bd700 con 0x7fb878031f30 2023-10-30T12:10:21.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.505+0000 7fb8870df640 1 librados: init done 2023-10-30T12:10:21.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.506+0000 7fb8870df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.506+0000 7fb8870df640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.506+0000 7fb8870df640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.506+0000 7fb8870df640 1 -- 172.21.15.3:0/164245000 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87802af40 con 0x7fb878031f30 2023-10-30T12:10:21.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.527+0000 7fb876ffd640 1 -- 172.21.15.3:0/164245000 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb880072040 con 0x7fb878031f30 2023-10-30T12:10:21.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.527+0000 7fb876ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.527+0000 7fb876ffd640 1 -- 172.21.15.3:0/164245000 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb880092400 con 0x7fb878031f30 2023-10-30T12:10:21.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.527+0000 7fb876ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.527+0000 7fb876ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.527+0000 7fb876ffd640 10 monclient: dump: 2023-10-30T12:10:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.572 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.527+0000 7fb8848da640 4 set_mon_vals no callback set 2023-10-30T12:10:21.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.528+0000 7fbbf6d9d640 1 --2- 172.21.15.3:0/4025230614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf81423f0 0x7fbbf814c320 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009d90 tx=0x7fbbf002bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41b01d091bc54994 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.541+0000 7f3dceffd640 1 -- 172.21.15.3:0/2538696770 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f3dd0011cb0 con 0x7f3de0141b30 2023-10-30T12:10:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.541+0000 7f3dceffd640 1 --2- 172.21.15.3:0/2538696770 >> v2:172.21.15.92:6800/1750155735 conn(0x7f3dc006c810 0x7f3dc006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.542+0000 7f3dde59c640 1 --2- 172.21.15.3:0/2538696770 >> v2:172.21.15.92:6800/1750155735 conn(0x7f3dc006c810 0x7f3dc006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.542+0000 7f3dde59c640 10 monclient: get_auth_request con 0x7f3dc006c810 auth_method 0 2023-10-30T12:10:21.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.542+0000 7f3dde59c640 1 --2- 172.21.15.3:0/2538696770 >> v2:172.21.15.92:6800/1750155735 conn(0x7f3dc006c810 0x7f3dc006ecd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4035000 tx=0x7f3dd4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.542+0000 7f3dceffd640 10 client.4336.objecter ms_handle_connect 0x7f3dc006c810 2023-10-30T12:10:21.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.548+0000 7f3dceffd640 1 -- 172.21.15.3:0/2538696770 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f3dd0058cc0 con 0x7f3de0141b30 2023-10-30T12:10:21.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.548+0000 7f3dceffd640 10 client.4336.objecter ms_dispatch 0x7f3de0000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:21.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.548+0000 7f3dceffd640 3 client.4336.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:21.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.548+0000 7f3dceffd640 3 client.4336.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:21.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.548+0000 7f3dceffd640 20 client.4336.objecter dump_active .. 0 homeless 2023-10-30T12:10:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.551+0000 7f9d8bfff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d8c141df0 0x7f9d8c142270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:60940/0 (socket says 172.21.15.3:60940) 2023-10-30T12:10:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.551+0000 7f9d8bfff640 1 -- 172.21.15.3:0/298916106 learned_addr learned my addr 172.21.15.3:0/298916106 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.551+0000 7f9d8bfff640 10 monclient(hunting): get_auth_request con 0x7f9d8c141df0 auth_method 0 2023-10-30T12:10:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.551+0000 7f9d8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.551+0000 7f9d8bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.551+0000 7f9d8bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.558+0000 7f3dceffd640 1 -- 172.21.15.3:0/2538696770 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3dd0017070 con 0x7f3de0141b30 2023-10-30T12:10:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.558+0000 7f3dceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.558+0000 7f3dceffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.559+0000 7ff7bffff640 1 -- 172.21.15.3:0/1611936587 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7ff7c8007460 con 0x7ff7d80b0990 2023-10-30T12:10:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.560+0000 7ff7bffff640 1 --2- 172.21.15.3:0/1611936587 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff7c406c7c0 0x7ff7c406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.560+0000 7ff7d77fe640 1 --2- 172.21.15.3:0/1611936587 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff7c406c7c0 0x7ff7c406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.560+0000 7ff7d77fe640 10 monclient: get_auth_request con 0x7ff7c406c7c0 auth_method 0 2023-10-30T12:10:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.561+0000 7ff7bffff640 10 client.4356.objecter ms_handle_connect 0x7ff7c406c7c0 2023-10-30T12:10:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.561+0000 7ff7d77fe640 1 --2- 172.21.15.3:0/1611936587 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff7c406c7c0 0x7ff7c406ec80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0000f80 tx=0x7ff7d0032000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.566+0000 7f9bfcff9640 1 -- 172.21.15.3:0/614844551 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f9bec03a040 con 0x7f9bf80cc960 2023-10-30T12:10:21.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.566+0000 7f9bfcff9640 1 --2- 172.21.15.3:0/614844551 >> v2:172.21.15.92:6800/1750155735 conn(0x7f9be406c840 0x7f9be406ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.567+0000 7f9bfffff640 1 --2- 172.21.15.3:0/614844551 >> v2:172.21.15.92:6800/1750155735 conn(0x7f9be406c840 0x7f9be406ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.567+0000 7f9bfffff640 10 monclient: get_auth_request con 0x7f9be406c840 auth_method 0 2023-10-30T12:10:21.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.567+0000 7f9bfffff640 1 --2- 172.21.15.3:0/614844551 >> v2:172.21.15.92:6800/1750155735 conn(0x7f9be406c840 0x7f9be406ed00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9c00000b60 tx=0x7f9c00074c70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.567+0000 7f9bfcff9640 10 client.4277.objecter ms_handle_connect 0x7f9be406c840 2023-10-30T12:10:21.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.576+0000 7ff7bffff640 1 -- 172.21.15.3:0/1611936587 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7ff7c80577b0 con 0x7ff7d80b0990 2023-10-30T12:10:21.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.576+0000 7ff7bffff640 10 client.4356.objecter ms_dispatch 0x7ff7d8000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:21.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.576+0000 7ff7bffff640 3 client.4356.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:21.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.576+0000 7ff7bffff640 3 client.4356.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:21.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.576+0000 7ff7bffff640 20 client.4356.objecter dump_active .. 0 homeless 2023-10-30T12:10:21.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.576+0000 7ff7bffff640 1 -- 172.21.15.3:0/1611936587 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff7c805b850 con 0x7ff7d80b0990 2023-10-30T12:10:21.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.576+0000 7ff7bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.576+0000 7ff7bffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.582+0000 7f9d8affd640 1 --2- 172.21.15.3:0/298916106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d8c0acc00 0x7f9d8c0ad040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.582+0000 7f9d8affd640 10 monclient(hunting): get_auth_request con 0x7f9d8c0acc00 auth_method 0 2023-10-30T12:10:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.582+0000 7f9d8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.582+0000 7f9d8affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.582+0000 7f9d8affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.583+0000 7f9d8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.583+0000 7f9d8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.583+0000 7f9d8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.585+0000 7f9bfcff9640 1 -- 172.21.15.3:0/614844551 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f9bec07a1d0 con 0x7f9bf80cc960 2023-10-30T12:10:21.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.585+0000 7f9bfcff9640 10 client.4277.objecter ms_dispatch 0x7f9c00000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:21.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.585+0000 7f9bfcff9640 3 client.4277.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:21.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.585+0000 7f9bfcff9640 3 client.4277.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:21.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.585+0000 7f9bfcff9640 20 client.4277.objecter dump_active .. 0 homeless 2023-10-30T12:10:21.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.585+0000 7f9bfcff9640 1 -- 172.21.15.3:0/614844551 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9bec079ad0 con 0x7f9bf80cc960 2023-10-30T12:10:21.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.585+0000 7f9bfcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.585+0000 7f9bfcff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:21.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.586+0000 7f9d8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.586+0000 7f9d8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.586+0000 7f9d8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.588+0000 7f9d8affd640 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2023-10-30T12:10:21.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.588+0000 7f9d8affd640 1 -- 172.21.15.3:0/298916106 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d8c141df0 msgr2=0x7f9d8c142270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.588+0000 7f9d8affd640 1 --2- 172.21.15.3:0/298916106 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d8c141df0 0x7f9d8c142270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.588+0000 7f9d8affd640 1 -- 172.21.15.3:0/298916106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d740053d0 msgr2=0x7f9d8c1cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.588+0000 7f9d8affd640 1 --2- 172.21.15.3:0/298916106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d740053d0 0x7f9d8c1cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.588+0000 7f9d8affd640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.588+0000 7f9d8affd640 1 monclient: found mon.noname-a 2023-10-30T12:10:21.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.588+0000 7f9d8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.588+0000 7f9d8affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:21.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.588+0000 7f9d8affd640 1 -- 172.21.15.3:0/298916106 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c1427b0 con 0x7f9d8c0acc00 2023-10-30T12:10:21.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.592+0000 7f9d8affd640 1 --2- 172.21.15.3:0/298916106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d8c0acc00 0x7f9d8c0ad040 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c007fd0 tx=0x7f9d7c007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.592+0000 7f9d73fff640 10 client.?.objecter ms_handle_connect 0x7f9d8c0acc00 2023-10-30T12:10:21.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.592+0000 7f9d73fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:21.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d73fff640 1 -- 172.21.15.3:0/298916106 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c00f990 con 0x7f9d8c0acc00 2023-10-30T12:10:21.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:21.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d73fff640 10 monclient: dump: 2023-10-30T12:10:21.599 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.599 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.600 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.600 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.600 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.600 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.600 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.601 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.601 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.601 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d73fff640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d73fff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d73fff640 1 -- 172.21.15.3:0/298916106 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7c00e040 con 0x7f9d8c0acc00 2023-10-30T12:10:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d73fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d73fff640 1 -- 172.21.15.3:0/298916106 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c00fd00 con 0x7f9d8c0acc00 2023-10-30T12:10:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:21.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d73fff640 10 monclient: dump: 2023-10-30T12:10:21.603 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.603 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.603 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.603 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.604 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.604 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.604 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.604 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.604 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.605 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d91a94640 5 monclient: authenticate success, global_id 4359 2023-10-30T12:10:21.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d91a94640 10 monclient: _renew_subs 2023-10-30T12:10:21.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d91a94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:21.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d91a94640 1 -- 172.21.15.3:0/298916106 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8c1da4e0 con 0x7f9d8c0acc00 2023-10-30T12:10:21.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d89ffb640 4 set_mon_vals no callback set 2023-10-30T12:10:21.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d91a94640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:21.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d91a94640 10 monclient: _renew_subs 2023-10-30T12:10:21.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d91a94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:21.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d91a94640 1 -- 172.21.15.3:0/298916106 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8c1da9b0 con 0x7f9d8c0acc00 2023-10-30T12:10:21.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.597+0000 7f9d91a94640 1 librados: init done 2023-10-30T12:10:21.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.598+0000 7f9d91a94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.598+0000 7f9d91a94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.598+0000 7f9d91a94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:21.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.598+0000 7f9d91a94640 1 -- 172.21.15.3:0/298916106 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8c04e9b0 con 0x7f9d8c0acc00 2023-10-30T12:10:21.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.744+0000 7f3de4f05640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T12:10:21.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.744+0000 7f3de4f05640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T12:10:21.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.744+0000 7f3de4f05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.745+0000 7f3de4f05640 1 -- 172.21.15.3:0/2538696770 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f3de013c880 con 0x7f3de0141b30 2023-10-30T12:10:21.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.764+0000 7ff7ddf5a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:10:21.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.764+0000 7ff7ddf5a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:10:21.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.764+0000 7ff7ddf5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:21.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.764+0000 7ff7ddf5a640 1 -- 172.21.15.3:0/1611936587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ff7d80ad160 con 0x7ff7d80b0990 2023-10-30T12:10:21.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.768+0000 7f9c067b0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T12:10:21.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.768+0000 7f9c067b0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T12:10:21.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.768+0000 7f9c067b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:21.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.768+0000 7f9c067b0640 1 -- 172.21.15.3:0/614844551 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f9bf8030c60 con 0x7f9bf80cc960 2023-10-30T12:10:21.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.772+0000 7f9bfcff9640 1 -- 172.21.15.3:0/614844551 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9bec07dc20 con 0x7f9bf80cc960 2023-10-30T12:10:21.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.773+0000 7f9bfcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T12:10:21.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.773+0000 7f9bfcff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:21.775 INFO:teuthology.orchestra.run.smithi003.stdout:60129542149 2023-10-30T12:10:21.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.774+0000 7f9c067b0640 10 librados: watch_flush enter 2023-10-30T12:10:21.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.774+0000 7f9c067b0640 10 librados: watch_flush exit 2023-10-30T12:10:21.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.774+0000 7f9c067b0640 20 client.4277.objecter shutdown clearing up homeless session... 2023-10-30T12:10:21.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.774+0000 7f9c067b0640 10 client.4277.objecter successfully canceled tick 2023-10-30T12:10:21.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.774+0000 7f9c067b0640 1 -- 172.21.15.3:0/614844551 >> v2:172.21.15.92:6800/1750155735 conn(0x7f9be406c840 msgr2=0x7f9be406ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.774+0000 7f9c067b0640 1 --2- 172.21.15.3:0/614844551 >> v2:172.21.15.92:6800/1750155735 conn(0x7f9be406c840 0x7f9be406ed00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9c00000b60 tx=0x7f9c00074c70 comp rx=0 tx=0).stop 2023-10-30T12:10:21.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.774+0000 7f9c067b0640 10 monclient: shutdown 2023-10-30T12:10:21.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.774+0000 7f9c067b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:21.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.774+0000 7f9c067b0640 1 -- 172.21.15.3:0/614844551 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9bf80cc960 msgr2=0x7f9bf80ca270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.774+0000 7f9c067b0640 1 --2- 172.21.15.3:0/614844551 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9bf80cc960 0x7f9bf80ca270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9bec02bdd0 tx=0x7f9bec0046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.775+0000 7f9c067b0640 1 -- 172.21.15.3:0/614844551 shutdown_connections 2023-10-30T12:10:21.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.775+0000 7f9c067b0640 1 --2- 172.21.15.3:0/614844551 >> v2:172.21.15.92:6800/1750155735 conn(0x7f9be406c840 0x7f9be406ed00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.775+0000 7f9c067b0640 1 --2- 172.21.15.3:0/614844551 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9bf80cc960 0x7f9bf80ca270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.775+0000 7f9c067b0640 1 --2- 172.21.15.3:0/614844551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bf8036280 0x7f9bf80ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.775+0000 7f9c067b0640 1 --2- 172.21.15.3:0/614844551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf80070a0 0x7f9bf80cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.775+0000 7f9c067b0640 1 -- 172.21.15.3:0/614844551 >> 172.21.15.3:0/614844551 conn(0x7f9bf8006040 msgr2=0x7f9bf8039720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:21.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.775+0000 7f9c067b0640 1 -- 172.21.15.3:0/614844551 shutdown_connections 2023-10-30T12:10:21.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.775+0000 7f9c067b0640 1 -- 172.21.15.3:0/614844551 wait complete. 2023-10-30T12:10:21.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.775+0000 7f9c067b0640 1 librados: shutdown 2023-10-30T12:10:21.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.7 2023-10-30T12:10:21.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63c27fc640 1 -- 172.21.15.3:0/1163036885 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0453f0 con 0x7f63c40c8310 2023-10-30T12:10:21.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63c27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:21.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63c27fc640 10 monclient: dump: 2023-10-30T12:10:21.793 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.793 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.793 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.793 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.793 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.794 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.794 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.794 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.794 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.794 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63c27fc640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63c27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63c27fc640 1 -- 172.21.15.3:0/1163036885 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc03f030 con 0x7f63c40c8310 2023-10-30T12:10:21.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63c27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63c27fc640 1 -- 172.21.15.3:0/1163036885 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc045800 con 0x7f63c40c8310 2023-10-30T12:10:21.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63d4026640 5 monclient: authenticate success, global_id 4342 2023-10-30T12:10:21.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63d4026640 10 monclient: _renew_subs 2023-10-30T12:10:21.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63d4026640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63d4026640 1 -- 172.21.15.3:0/1163036885 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c402e470 con 0x7f63c40c8310 2023-10-30T12:10:21.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63c3fff640 4 set_mon_vals no callback set 2023-10-30T12:10:21.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:21.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63d4026640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:21.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63c27fc640 10 monclient: dump: 2023-10-30T12:10:21.798 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.798 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.798 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.798 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.798 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.799 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.799 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.799 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.799 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.799 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63d4026640 10 monclient: _renew_subs 2023-10-30T12:10:21.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63d4026640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63d4026640 1 -- 172.21.15.3:0/1163036885 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c402ee90 con 0x7f63c40c8310 2023-10-30T12:10:21.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.790+0000 7f63d4026640 1 librados: init done 2023-10-30T12:10:21.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7f63d4026640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7f63d4026640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7f63d4026640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7f63d4026640 1 -- 172.21.15.3:0/1163036885 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c40c5860 con 0x7f63c40c8310 2023-10-30T12:10:21.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff99f7fe640 1 -- 172.21.15.3:0/3554776335 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b000ce70 con 0x7ff9b8144f50 2023-10-30T12:10:21.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff99f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff99f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:21.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff99f7fe640 10 monclient: dump: 2023-10-30T12:10:21.803 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.803 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.803 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.804 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.804 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.804 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.804 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.804 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.805 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.805 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff99f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff99f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff99f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff9bd42d640 5 monclient: authenticate success, global_id 4345 2023-10-30T12:10:21.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff99f7fe640 1 -- 172.21.15.3:0/3554776335 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b0003990 con 0x7ff9b8144f50 2023-10-30T12:10:21.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff99f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff99f7fe640 1 -- 172.21.15.3:0/3554776335 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b0003d10 con 0x7ff9b8144f50 2023-10-30T12:10:21.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff99f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff99f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:21.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff99f7fe640 10 monclient: dump: 2023-10-30T12:10:21.807 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:21.807 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:21.807 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.807 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:21.807 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.808 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:21.808 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:21.808 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:21.808 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:21.808 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:21.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff9b57fa640 4 set_mon_vals no callback set 2023-10-30T12:10:21.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff9bd42d640 10 monclient: _renew_subs 2023-10-30T12:10:21.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff9bd42d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff9bd42d640 1 -- 172.21.15.3:0/3554776335 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b81377f0 con 0x7ff9b8144f50 2023-10-30T12:10:21.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff9bd42d640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:21.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff9bd42d640 10 monclient: _renew_subs 2023-10-30T12:10:21.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff9bd42d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff9bd42d640 1 -- 172.21.15.3:0/3554776335 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b8137cc0 con 0x7ff9b8144f50 2023-10-30T12:10:21.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.791+0000 7ff9bd42d640 1 librados: init done 2023-10-30T12:10:21.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.792+0000 7ff9bd42d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.792+0000 7ff9bd42d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.792+0000 7ff9bd42d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:21.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.792+0000 7ff9bd42d640 1 -- 172.21.15.3:0/3554776335 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b804e9b0 con 0x7ff9b8144f50 2023-10-30T12:10:21.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.880+0000 7f9d73fff640 1 -- 172.21.15.3:0/298916106 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f9d7c01d460 con 0x7f9d8c0acc00 2023-10-30T12:10:21.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.881+0000 7f9d73fff640 1 --2- 172.21.15.3:0/298916106 >> v2:172.21.15.92:6800/1750155735 conn(0x7f9d7806c7c0 0x7f9d7806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.881+0000 7f9d8b7fe640 1 --2- 172.21.15.3:0/298916106 >> v2:172.21.15.92:6800/1750155735 conn(0x7f9d7806c7c0 0x7f9d7806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.881+0000 7f9d8b7fe640 10 monclient: get_auth_request con 0x7f9d7806c7c0 auth_method 0 2023-10-30T12:10:21.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.881+0000 7f9d8b7fe640 1 --2- 172.21.15.3:0/298916106 >> v2:172.21.15.92:6800/1750155735 conn(0x7f9d7806c7c0 0x7f9d7806ec80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9d840070b0 tx=0x7f9d84007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.881+0000 7f9d73fff640 10 client.4359.objecter ms_handle_connect 0x7f9d7806c7c0 2023-10-30T12:10:21.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.895+0000 7f9d73fff640 1 -- 172.21.15.3:0/298916106 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f9d7c00fe70 con 0x7f9d8c0acc00 2023-10-30T12:10:21.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.895+0000 7f9d73fff640 10 client.4359.objecter ms_dispatch 0x7f9d8c000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:21.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.895+0000 7f9d73fff640 3 client.4359.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:21.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.895+0000 7f9d73fff640 3 client.4359.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:21.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.895+0000 7f9d73fff640 20 client.4359.objecter dump_active .. 0 homeless 2023-10-30T12:10:21.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.895+0000 7f9d73fff640 1 -- 172.21.15.3:0/298916106 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9d7c018070 con 0x7f9d8c0acc00 2023-10-30T12:10:21.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.895+0000 7f9d73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.895+0000 7f9d73fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:21.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.905+0000 7ff7bffff640 1 -- 172.21.15.3:0/1611936587 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff7c805fa00 con 0x7ff7d80b0990 2023-10-30T12:10:21.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.905+0000 7ff7bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:10:21.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.905+0000 7ff7bffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:21.907 INFO:teuthology.orchestra.run.smithi003.stdout:68719476742 2023-10-30T12:10:21.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 10 librados: watch_flush enter 2023-10-30T12:10:21.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 10 librados: watch_flush exit 2023-10-30T12:10:21.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 20 client.4356.objecter shutdown clearing up homeless session... 2023-10-30T12:10:21.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 10 client.4356.objecter successfully canceled tick 2023-10-30T12:10:21.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 1 -- 172.21.15.3:0/1611936587 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff7c406c7c0 msgr2=0x7ff7c406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 1 --2- 172.21.15.3:0/1611936587 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff7c406c7c0 0x7ff7c406ec80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0000f80 tx=0x7ff7d0032000 comp rx=0 tx=0).stop 2023-10-30T12:10:21.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 10 monclient: shutdown 2023-10-30T12:10:21.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:21.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 1 -- 172.21.15.3:0/1611936587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d80b0990 msgr2=0x7ff7d81d7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 1 --2- 172.21.15.3:0/1611936587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d80b0990 0x7ff7d81d7400 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800c3f0 tx=0x7ff7c800c8b0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 1 -- 172.21.15.3:0/1611936587 shutdown_connections 2023-10-30T12:10:21.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 1 --2- 172.21.15.3:0/1611936587 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff7c406c7c0 0x7ff7c406ec80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 1 --2- 172.21.15.3:0/1611936587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d80b0990 0x7ff7d81d7400 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 1 --2- 172.21.15.3:0/1611936587 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7d806f2c0 0x7ff7d81d6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 1 --2- 172.21.15.3:0/1611936587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7d806eea0 0x7ff7d80acc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 1 -- 172.21.15.3:0/1611936587 >> 172.21.15.3:0/1611936587 conn(0x7ff7d80b3c20 msgr2=0x7ff7d806e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:21.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 1 -- 172.21.15.3:0/1611936587 shutdown_connections 2023-10-30T12:10:21.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 1 -- 172.21.15.3:0/1611936587 wait complete. 2023-10-30T12:10:21.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:21.907+0000 7ff7ddf5a640 1 librados: shutdown 2023-10-30T12:10:21.919 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476742 for osd.2 2023-10-30T12:10:22.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.001+0000 7fbbf559a640 1 -- 172.21.15.3:0/4025230614 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf002cb30 con 0x7fbbf81423f0 2023-10-30T12:10:22.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.001+0000 7fbbf559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.001+0000 7fbbf559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:22.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.001+0000 7fbbf559a640 10 monclient: dump: 2023-10-30T12:10:22.003 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:22.003 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:22.004 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.004 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.004 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.004 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:22.004 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:22.005 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:22.005 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:22.005 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:22.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.001+0000 7fbbf559a640 10 monclient: _finish_auth 0 2023-10-30T12:10:22.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.001+0000 7fbbf559a640 10 monclient: _check_auth_tickets 2023-10-30T12:10:22.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.001+0000 7fbbf559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:22.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.001+0000 7fbbfc997640 5 monclient: authenticate success, global_id 4330 2023-10-30T12:10:22.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.001+0000 7fbbfc997640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:22.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.081+0000 7f9d91a94640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T12:10:22.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.081+0000 7f9d91a94640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T12:10:22.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.081+0000 7f9d91a94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:22.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.081+0000 7f9d91a94640 1 -- 172.21.15.3:0/298916106 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9d8c1427b0 con 0x7f9d8c0acc00 2023-10-30T12:10:22.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.086+0000 7f9d73fff640 1 -- 172.21.15.3:0/298916106 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9d7c058eb0 con 0x7f9d8c0acc00 2023-10-30T12:10:22.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.086+0000 7f9d73fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T12:10:22.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.086+0000 7f9d73fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:22.088 INFO:teuthology.orchestra.run.smithi003.stdout:68719476740 2023-10-30T12:10:22.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 10 librados: watch_flush enter 2023-10-30T12:10:22.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 10 librados: watch_flush exit 2023-10-30T12:10:22.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 20 client.4359.objecter shutdown clearing up homeless session... 2023-10-30T12:10:22.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 10 client.4359.objecter successfully canceled tick 2023-10-30T12:10:22.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 1 -- 172.21.15.3:0/298916106 >> v2:172.21.15.92:6800/1750155735 conn(0x7f9d7806c7c0 msgr2=0x7f9d7806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 1 --2- 172.21.15.3:0/298916106 >> v2:172.21.15.92:6800/1750155735 conn(0x7f9d7806c7c0 0x7f9d7806ec80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9d840070b0 tx=0x7f9d84007040 comp rx=0 tx=0).stop 2023-10-30T12:10:22.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 10 monclient: shutdown 2023-10-30T12:10:22.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 1 -- 172.21.15.3:0/298916106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d8c0acc00 msgr2=0x7f9d8c0ad040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 1 --2- 172.21.15.3:0/298916106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d8c0acc00 0x7f9d8c0ad040 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c007fd0 tx=0x7f9d7c007720 comp rx=0 tx=0).stop 2023-10-30T12:10:22.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 1 -- 172.21.15.3:0/298916106 shutdown_connections 2023-10-30T12:10:22.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 1 --2- 172.21.15.3:0/298916106 >> v2:172.21.15.92:6800/1750155735 conn(0x7f9d7806c7c0 0x7f9d7806ec80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 1 --2- 172.21.15.3:0/298916106 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d8c141df0 0x7f9d8c142270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 1 --2- 172.21.15.3:0/298916106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d8c0acc00 0x7f9d8c0ad040 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 1 --2- 172.21.15.3:0/298916106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d740053d0 0x7f9d8c1cbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 1 -- 172.21.15.3:0/298916106 >> 172.21.15.3:0/298916106 conn(0x7f9d8c065ea0 msgr2=0x7f9d8c0737b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 1 -- 172.21.15.3:0/298916106 shutdown_connections 2023-10-30T12:10:22.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 1 -- 172.21.15.3:0/298916106 wait complete. 2023-10-30T12:10:22.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.088+0000 7f9d91a94640 1 librados: shutdown 2023-10-30T12:10:22.100 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.0 2023-10-30T12:10:22.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.248+0000 7fbbf559a640 1 -- 172.21.15.3:0/4025230614 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf002cca0 con 0x7fbbf81423f0 2023-10-30T12:10:22.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.248+0000 7fbbf559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:22.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.248+0000 7fbbf559a640 1 -- 172.21.15.3:0/4025230614 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0034660 con 0x7fbbf81423f0 2023-10-30T12:10:22.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.248+0000 7fbbf559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.248+0000 7fbbf559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:22.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.248+0000 7fbbf559a640 10 monclient: dump: 2023-10-30T12:10:22.251 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:22.251 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:22.251 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.251 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.252 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.252 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:22.252 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:22.252 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:22.252 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:22.253 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:22.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.248+0000 7fbbfc997640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:22.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.248+0000 7fbbfc997640 4 set_mon_vals no callback set 2023-10-30T12:10:22.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.249+0000 7f3dceffd640 1 -- 172.21.15.3:0/2538696770 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3dd00585c0 con 0x7f3de0141b30 2023-10-30T12:10:22.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.249+0000 7f3dceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T12:10:22.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.249+0000 7f3dceffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:22.254 INFO:teuthology.orchestra.run.smithi003.stdout:55834574851 2023-10-30T12:10:22.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.249+0000 7fbbfc997640 10 monclient: shutdown 2023-10-30T12:10:22.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.249+0000 7fbbfc997640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.249+0000 7fbbfc997640 1 -- 172.21.15.3:0/4025230614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf81423f0 msgr2=0x7fbbf814c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.249+0000 7fbbfc997640 1 --2- 172.21.15.3:0/4025230614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf81423f0 0x7fbbf814c320 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009d90 tx=0x7fbbf002bd80 comp rx=0 tx=0).stop 2023-10-30T12:10:22.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.249+0000 7fbbfc997640 1 -- 172.21.15.3:0/4025230614 shutdown_connections 2023-10-30T12:10:22.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.249+0000 7fbbfc997640 1 --2- 172.21.15.3:0/4025230614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf81423f0 0x7fbbf814c320 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.249+0000 7fbbfc997640 1 --2- 172.21.15.3:0/4025230614 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf80b13c0 0x7fbbf8141eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.249+0000 7fbbfc997640 1 -- 172.21.15.3:0/4025230614 >> 172.21.15.3:0/4025230614 conn(0x7fbbf80b7180 msgr2=0x7fbbf80b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.249+0000 7fbbfc997640 1 -- 172.21.15.3:0/4025230614 shutdown_connections 2023-10-30T12:10:22.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.249+0000 7fbbfc997640 1 -- 172.21.15.3:0/4025230614 wait complete. 2023-10-30T12:10:22.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.250+0000 7fbbfc997640 10 monclient: build_initial_monmap 2023-10-30T12:10:22.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.250+0000 7fbbfc997640 10 monclient: monmap: 2023-10-30T12:10:22.258 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:22.258 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:22.258 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:22.259 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:22.259 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:22.259 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:22.259 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:22.259 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:22.260 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:22.260 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:22.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.250+0000 7fbbfc997640 1 librados: starting msgr at 2023-10-30T12:10:22.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.250+0000 7fbbfc997640 1 librados: starting objecter 2023-10-30T12:10:22.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.250+0000 7fbbfc997640 1 Processor -- start 2023-10-30T12:10:22.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.250+0000 7fbbfc997640 1 -- start start 2023-10-30T12:10:22.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.250+0000 7fbbfc997640 1 librados: setting wanted keys 2023-10-30T12:10:22.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.250+0000 7fbbfc997640 1 librados: calling monclient init 2023-10-30T12:10:22.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.250+0000 7fbbfc997640 10 monclient: init 2023-10-30T12:10:22.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.250+0000 7f3de4f05640 10 librados: watch_flush enter 2023-10-30T12:10:22.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 10 librados: watch_flush exit 2023-10-30T12:10:22.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 20 client.4336.objecter shutdown clearing up homeless session... 2023-10-30T12:10:22.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 10 client.4336.objecter successfully canceled tick 2023-10-30T12:10:22.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 1 -- 172.21.15.3:0/2538696770 >> v2:172.21.15.92:6800/1750155735 conn(0x7f3dc006c810 msgr2=0x7f3dc006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 1 --2- 172.21.15.3:0/2538696770 >> v2:172.21.15.92:6800/1750155735 conn(0x7f3dc006c810 0x7f3dc006ecd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4035000 tx=0x7f3dd4030000 comp rx=0 tx=0).stop 2023-10-30T12:10:22.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 10 monclient: shutdown 2023-10-30T12:10:22.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 1 -- 172.21.15.3:0/2538696770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3de0141b30 msgr2=0x7f3de013f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 1 --2- 172.21.15.3:0/2538696770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3de0141b30 0x7f3de013f2e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000c430 tx=0x7f3dd000c8f0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 1 -- 172.21.15.3:0/2538696770 shutdown_connections 2023-10-30T12:10:22.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 1 --2- 172.21.15.3:0/2538696770 >> v2:172.21.15.92:6800/1750155735 conn(0x7f3dc006c810 0x7f3dc006ecd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 1 --2- 172.21.15.3:0/2538696770 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3de0141f50 0x7f3de00abb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 1 --2- 172.21.15.3:0/2538696770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3de0141b30 0x7f3de013f2e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 1 --2- 172.21.15.3:0/2538696770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3de0071180 0x7f3de013cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 1 -- 172.21.15.3:0/2538696770 >> 172.21.15.3:0/2538696770 conn(0x7f3de00b3a10 msgr2=0x7f3de00737b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 1 -- 172.21.15.3:0/2538696770 shutdown_connections 2023-10-30T12:10:22.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 1 -- 172.21.15.3:0/2538696770 wait complete. 2023-10-30T12:10:22.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7f3de4f05640 1 librados: shutdown 2023-10-30T12:10:22.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:22.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:10:22.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf80a8e40 0x7fbbf81ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 10 monclient(hunting): picked mon.noname-b con 0x7fbbf80a8e40 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:22.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf81423f0 0x7fbbf81d7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 10 monclient(hunting): picked mon.noname-c con 0x7fbbf81423f0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:22.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbf80ae540 0x7fbbf80af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 10 monclient(hunting): picked mon.noname-a con 0x7fbbf80ae540 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:22.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf80aa3e0 con 0x7fbbf80ae540 2023-10-30T12:10:22.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf81d6c60 con 0x7fbbf80a8e40 2023-10-30T12:10:22.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf81d6f20 con 0x7fbbf81423f0 2023-10-30T12:10:22.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:22.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.251+0000 7fbbfc997640 10 monclient(hunting): authenticate will time out at 1413.465599s 2023-10-30T12:10:22.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.252+0000 7fbbf6d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbf80ae540 0x7fbbf80af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.252+0000 7fbbf5d9b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf81423f0 0x7fbbf81d7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.252+0000 7fbbf5d9b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf81423f0 0x7fbbf81d7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:48400/0 (socket says 172.21.15.3:48400) 2023-10-30T12:10:22.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.252+0000 7fbbf5d9b640 1 -- 172.21.15.3:0/4200346692 learned_addr learned my addr 172.21.15.3:0/4200346692 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:22.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.252+0000 7fbbf5d9b640 10 monclient(hunting): get_auth_request con 0x7fbbf81423f0 auth_method 0 2023-10-30T12:10:22.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.252+0000 7fbbf5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.252+0000 7fbbf5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.252+0000 7fbbf5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.252+0000 7fbbf6d9d640 10 monclient(hunting): get_auth_request con 0x7fbbf80ae540 auth_method 0 2023-10-30T12:10:22.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.252+0000 7fbbf6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.252+0000 7fbbf6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.252+0000 7fbbf6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.253+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.253+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.253+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.253+0000 7fbbf5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.253+0000 7fbbf5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.253+0000 7fbbf5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.256+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2023-10-30T12:10:22.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.256+0000 7fbbf6d9d640 1 -- 172.21.15.3:0/4200346692 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf81423f0 msgr2=0x7fbbf81d7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.256+0000 7fbbf6d9d640 1 --2- 172.21.15.3:0/4200346692 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf81423f0 0x7fbbf81d7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.256+0000 7fbbf6d9d640 1 -- 172.21.15.3:0/4200346692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf80a8e40 msgr2=0x7fbbf81ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.256+0000 7fbbf6d9d640 1 --2- 172.21.15.3:0/4200346692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf80a8e40 0x7fbbf81ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.256+0000 7fbbf6d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:10:22.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.256+0000 7fbbf6d9d640 1 monclient: found mon.noname-a 2023-10-30T12:10:22.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.256+0000 7fbbf6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:22.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.256+0000 7fbbf6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:22.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.256+0000 7fbbf6d9d640 1 -- 172.21.15.3:0/4200346692 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf81d7980 con 0x7fbbf80ae540 2023-10-30T12:10:22.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.259+0000 7fbbf6d9d640 1 --2- 172.21.15.3:0/4200346692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbf80ae540 0x7fbbf80af2e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0033000 tx=0x7fbbf0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.259+0000 7fbbe6ffd640 10 client.?.objecter ms_handle_connect 0x7fbbf80ae540 2023-10-30T12:10:22.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.259+0000 7fbbe6ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:22.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.262+0000 7fb876ffd640 1 -- 172.21.15.3:0/164245000 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fb880092650 con 0x7fb878031f30 2023-10-30T12:10:22.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.262+0000 7fb876ffd640 1 --2- 172.21.15.3:0/164245000 >> v2:172.21.15.92:6800/1750155735 conn(0x7fb86406c7c0 0x7fb86406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.262+0000 7fb8860dd640 1 --2- 172.21.15.3:0/164245000 >> v2:172.21.15.92:6800/1750155735 conn(0x7fb86406c7c0 0x7fb86406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.262+0000 7fb8860dd640 10 monclient: get_auth_request con 0x7fb86406c7c0 auth_method 0 2023-10-30T12:10:22.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.263+0000 7fb8860dd640 1 --2- 172.21.15.3:0/164245000 >> v2:172.21.15.92:6800/1750155735 conn(0x7fb86406c7c0 0x7fb86406ec80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb87c037000 tx=0x7fb87c031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.263+0000 7fb876ffd640 10 client.4333.objecter ms_handle_connect 0x7fb86406c7c0 2023-10-30T12:10:22.281 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574851 for osd.6 2023-10-30T12:10:22.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbe6ffd640 1 -- 172.21.15.3:0/4200346692 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0040650 con 0x7fbbf80ae540 2023-10-30T12:10:22.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbe6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:22.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbe6ffd640 10 monclient: dump: 2023-10-30T12:10:22.283 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:22.283 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:22.283 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.283 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.284 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.284 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:22.284 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:22.284 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:22.284 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:22.285 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:22.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbe6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:10:22.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbe6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:22.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbe6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:22.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbe6ffd640 1 -- 172.21.15.3:0/4200346692 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf00407c0 con 0x7fbbf80ae540 2023-10-30T12:10:22.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbe6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:22.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbe6ffd640 1 -- 172.21.15.3:0/4200346692 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0040b80 con 0x7fbbf80ae540 2023-10-30T12:10:22.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbe6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:22.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbe6ffd640 10 monclient: dump: 2023-10-30T12:10:22.287 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:22.287 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:22.287 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.287 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.287 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.288 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:22.288 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:22.288 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:22.288 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:22.288 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:22.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbfc997640 5 monclient: authenticate success, global_id 4362 2023-10-30T12:10:22.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbf4d99640 4 set_mon_vals no callback set 2023-10-30T12:10:22.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbfc997640 10 monclient: _renew_subs 2023-10-30T12:10:22.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbfc997640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:22.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbfc997640 1 -- 172.21.15.3:0/4200346692 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf80af820 con 0x7fbbf80ae540 2023-10-30T12:10:22.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbfc997640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:22.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbfc997640 10 monclient: _renew_subs 2023-10-30T12:10:22.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbfc997640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:22.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbfc997640 1 -- 172.21.15.3:0/4200346692 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf81da160 con 0x7fbbf80ae540 2023-10-30T12:10:22.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.264+0000 7fbbfc997640 1 librados: init done 2023-10-30T12:10:22.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.265+0000 7fbbfc997640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.265+0000 7fbbfc997640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.265+0000 7fbbfc997640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:22.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.265+0000 7fbbfc997640 1 -- 172.21.15.3:0/4200346692 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf80ab610 con 0x7fbbf80ae540 2023-10-30T12:10:22.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.266+0000 7fb876ffd640 1 -- 172.21.15.3:0/164245000 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fb8800d1820 con 0x7fb878031f30 2023-10-30T12:10:22.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.266+0000 7fb876ffd640 10 client.4333.objecter ms_dispatch 0x7fb880000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:22.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.266+0000 7fb876ffd640 3 client.4333.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:22.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.266+0000 7fb876ffd640 3 client.4333.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:22.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.266+0000 7fb876ffd640 20 client.4333.objecter dump_active .. 0 homeless 2023-10-30T12:10:22.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.278+0000 7ff99f7fe640 1 -- 172.21.15.3:0/3554776335 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7ff9b0003e80 con 0x7ff9b8144f50 2023-10-30T12:10:22.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.278+0000 7ff99f7fe640 1 --2- 172.21.15.3:0/3554776335 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff9a406c7f0 0x7ff9a406ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.279+0000 7ff9b6ffd640 1 --2- 172.21.15.3:0/3554776335 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff9a406c7f0 0x7ff9a406ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.279+0000 7ff9b6ffd640 10 monclient: get_auth_request con 0x7ff9a406c7f0 auth_method 0 2023-10-30T12:10:22.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.279+0000 7ff99f7fe640 10 client.4345.objecter ms_handle_connect 0x7ff9a406c7f0 2023-10-30T12:10:22.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.279+0000 7ff9b6ffd640 1 --2- 172.21.15.3:0/3554776335 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff9a406c7f0 0x7ff9a406ecb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8035000 tx=0x7ff9a8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.280+0000 7ff99f7fe640 1 -- 172.21.15.3:0/3554776335 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7ff9b0058e00 con 0x7ff9b8144f50 2023-10-30T12:10:22.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.280+0000 7ff99f7fe640 10 client.4345.objecter ms_dispatch 0x7ff9b8000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:22.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.280+0000 7ff99f7fe640 3 client.4345.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:22.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.280+0000 7ff99f7fe640 3 client.4345.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:22.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.280+0000 7ff99f7fe640 20 client.4345.objecter dump_active .. 0 homeless 2023-10-30T12:10:22.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.285+0000 7fbbe6ffd640 1 -- 172.21.15.3:0/4200346692 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fbbf002cb40 con 0x7fbbf80ae540 2023-10-30T12:10:22.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.285+0000 7fbbe6ffd640 1 --2- 172.21.15.3:0/4200346692 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbbd806c810 0x7fbbd806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.286+0000 7fbbf659c640 1 --2- 172.21.15.3:0/4200346692 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbbd806c810 0x7fbbd806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.286+0000 7fbbf659c640 10 monclient: get_auth_request con 0x7fbbd806c810 auth_method 0 2023-10-30T12:10:22.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.286+0000 7fbbe6ffd640 10 client.4362.objecter ms_handle_connect 0x7fbbd806c810 2023-10-30T12:10:22.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.286+0000 7fbbf659c640 1 --2- 172.21.15.3:0/4200346692 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbbd806c810 0x7fbbd806ecd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80015f0 tx=0x7fbbe8007a40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.288+0000 7fbbe6ffd640 1 -- 172.21.15.3:0/4200346692 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fbbf007b460 con 0x7fbbf80ae540 2023-10-30T12:10:22.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.288+0000 7fbbe6ffd640 10 client.4362.objecter ms_dispatch 0x7fbbf8000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:22.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.288+0000 7fbbe6ffd640 3 client.4362.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:22.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.288+0000 7fbbe6ffd640 3 client.4362.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:22.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.288+0000 7fbbe6ffd640 20 client.4362.objecter dump_active .. 0 homeless 2023-10-30T12:10:22.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.292+0000 7fb876ffd640 1 -- 172.21.15.3:0/164245000 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb8800d58c0 con 0x7fb878031f30 2023-10-30T12:10:22.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.292+0000 7fb876ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.292+0000 7fb876ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:22.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.305+0000 7ff99f7fe640 1 -- 172.21.15.3:0/3554776335 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff9b0058700 con 0x7ff9b8144f50 2023-10-30T12:10:22.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.305+0000 7ff99f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.305+0000 7ff99f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:22.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.320+0000 7f63c27fc640 1 -- 172.21.15.3:0/1163036885 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f63bc0443a0 con 0x7f63c40c8310 2023-10-30T12:10:22.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.320+0000 7f63c27fc640 1 --2- 172.21.15.3:0/1163036885 >> v2:172.21.15.92:6800/1750155735 conn(0x7f63b006c810 0x7f63b006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.320+0000 7f63d1dc2640 1 --2- 172.21.15.3:0/1163036885 >> v2:172.21.15.92:6800/1750155735 conn(0x7f63b006c810 0x7f63b006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.320+0000 7f63d1dc2640 10 monclient: get_auth_request con 0x7f63b006c810 auth_method 0 2023-10-30T12:10:22.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.321+0000 7f63d1dc2640 1 --2- 172.21.15.3:0/1163036885 >> v2:172.21.15.92:6800/1750155735 conn(0x7f63b006c810 0x7f63b006ecd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009a40 tx=0x7f63c80091c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.321+0000 7f63c27fc640 10 client.4342.objecter ms_handle_connect 0x7f63b006c810 2023-10-30T12:10:22.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.325+0000 7f63c27fc640 1 -- 172.21.15.3:0/1163036885 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f63bc053080 con 0x7f63c40c8310 2023-10-30T12:10:22.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.325+0000 7f63c27fc640 10 client.4342.objecter ms_dispatch 0x7f63cc000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:22.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.325+0000 7f63c27fc640 3 client.4342.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:22.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.325+0000 7f63c27fc640 3 client.4342.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:22.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.325+0000 7f63c27fc640 20 client.4342.objecter dump_active .. 0 homeless 2023-10-30T12:10:22.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.334+0000 7f63c27fc640 1 -- 172.21.15.3:0/1163036885 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f63c402ee90 con 0x7f63c40c8310 2023-10-30T12:10:22.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.334+0000 7f63c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.334+0000 7f63c27fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:22.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.510+0000 7fb8870df640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:10:22.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.510+0000 7fb8870df640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:10:22.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.510+0000 7fb8870df640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:22.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.510+0000 7fb8870df640 1 -- 172.21.15.3:0/164245000 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb87815b320 con 0x7fb878031f30 2023-10-30T12:10:22.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.515+0000 7fb876ffd640 1 -- 172.21.15.3:0/164245000 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb8800d9a70 con 0x7fb878031f30 2023-10-30T12:10:22.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.515+0000 7fb876ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:10:22.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.515+0000 7fb876ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:22.518 INFO:teuthology.orchestra.run.smithi003.stdout:55834574851 2023-10-30T12:10:22.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 10 librados: watch_flush enter 2023-10-30T12:10:22.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 10 librados: watch_flush exit 2023-10-30T12:10:22.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 20 client.4333.objecter shutdown clearing up homeless session... 2023-10-30T12:10:22.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 10 client.4333.objecter successfully canceled tick 2023-10-30T12:10:22.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 1 -- 172.21.15.3:0/164245000 >> v2:172.21.15.92:6800/1750155735 conn(0x7fb86406c7c0 msgr2=0x7fb86406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 1 --2- 172.21.15.3:0/164245000 >> v2:172.21.15.92:6800/1750155735 conn(0x7fb86406c7c0 0x7fb86406ec80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb87c037000 tx=0x7fb87c031000 comp rx=0 tx=0).stop 2023-10-30T12:10:22.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 10 monclient: shutdown 2023-10-30T12:10:22.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 1 -- 172.21.15.3:0/164245000 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb878031f30 msgr2=0x7fb87815ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 1 --2- 172.21.15.3:0/164245000 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb878031f30 0x7fb87815ade0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb880067f90 tx=0x7fb880067340 comp rx=0 tx=0).stop 2023-10-30T12:10:22.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 1 -- 172.21.15.3:0/164245000 shutdown_connections 2023-10-30T12:10:22.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 1 --2- 172.21.15.3:0/164245000 >> v2:172.21.15.92:6800/1750155735 conn(0x7fb86406c7c0 0x7fb86406ec80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 1 --2- 172.21.15.3:0/164245000 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb878031f30 0x7fb87815ade0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 1 --2- 172.21.15.3:0/164245000 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb8780c3d00 0x7fb8780c0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 1 --2- 172.21.15.3:0/164245000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb878034aa0 0x7fb8780c3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 1 -- 172.21.15.3:0/164245000 >> 172.21.15.3:0/164245000 conn(0x7fb8780060a0 msgr2=0x7fb878032590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 1 -- 172.21.15.3:0/164245000 shutdown_connections 2023-10-30T12:10:22.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 1 -- 172.21.15.3:0/164245000 wait complete. 2023-10-30T12:10:22.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.517+0000 7fb8870df640 1 librados: shutdown 2023-10-30T12:10:22.530 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574851 for osd.5 2023-10-30T12:10:22.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.547+0000 7ff9bd42d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T12:10:22.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.547+0000 7ff9bd42d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T12:10:22.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.547+0000 7ff9bd42d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:22.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.547+0000 7ff9bd42d640 1 -- 172.21.15.3:0/3554776335 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff9b8137cc0 con 0x7ff9b8144f50 2023-10-30T12:10:22.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.550+0000 7ff99f7fe640 1 -- 172.21.15.3:0/3554776335 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff9b005c850 con 0x7ff9b8144f50 2023-10-30T12:10:22.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.550+0000 7ff99f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T12:10:22.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.550+0000 7ff99f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:22.552 INFO:teuthology.orchestra.run.smithi003.stdout:68719476740 2023-10-30T12:10:22.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 10 librados: watch_flush enter 2023-10-30T12:10:22.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 10 librados: watch_flush exit 2023-10-30T12:10:22.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 20 client.4345.objecter shutdown clearing up homeless session... 2023-10-30T12:10:22.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 10 client.4345.objecter successfully canceled tick 2023-10-30T12:10:22.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 1 -- 172.21.15.3:0/3554776335 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff9a406c7f0 msgr2=0x7ff9a406ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 1 --2- 172.21.15.3:0/3554776335 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff9a406c7f0 0x7ff9a406ecb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8035000 tx=0x7ff9a8008030 comp rx=0 tx=0).stop 2023-10-30T12:10:22.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 10 monclient: shutdown 2023-10-30T12:10:22.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 1 -- 172.21.15.3:0/3554776335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9b8144f50 msgr2=0x7ff9b80b7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 1 --2- 172.21.15.3:0/3554776335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9b8144f50 0x7ff9b80b7bf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000cc60 tx=0x7ff9b000a5c0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 1 -- 172.21.15.3:0/3554776335 shutdown_connections 2023-10-30T12:10:22.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 1 --2- 172.21.15.3:0/3554776335 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff9a406c7f0 0x7ff9a406ecb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 1 --2- 172.21.15.3:0/3554776335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9b8144f50 0x7ff9b80b7bf0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 1 --2- 172.21.15.3:0/3554776335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b813eb50 0x7ff9b80b5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 1 --2- 172.21.15.3:0/3554776335 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9b80b4350 0x7ff9b80b53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 1 -- 172.21.15.3:0/3554776335 >> 172.21.15.3:0/3554776335 conn(0x7ff9b8065ea0 msgr2=0x7ff9b81432e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 1 -- 172.21.15.3:0/3554776335 shutdown_connections 2023-10-30T12:10:22.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.552+0000 7ff9bd42d640 1 -- 172.21.15.3:0/3554776335 wait complete. 2023-10-30T12:10:22.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.553+0000 7ff9bd42d640 1 librados: shutdown 2023-10-30T12:10:22.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.3 2023-10-30T12:10:22.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.575+0000 7f63d4026640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:10:22.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.575+0000 7f63d4026640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:10:22.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.575+0000 7f63d4026640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:22.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.575+0000 7f63d4026640 1 -- 172.21.15.3:0/1163036885 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f63c4036590 con 0x7f63c40c8310 2023-10-30T12:10:22.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.577+0000 7fbbe6ffd640 1 -- 172.21.15.3:0/4200346692 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbbf007ad60 con 0x7fbbf80ae540 2023-10-30T12:10:22.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.577+0000 7fbbe6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.577+0000 7fbbe6ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:22.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.579+0000 7f63c27fc640 1 -- 172.21.15.3:0/1163036885 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f63bc07af60 con 0x7f63c40c8310 2023-10-30T12:10:22.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.579+0000 7f63c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:10:22.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.579+0000 7f63c27fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:22.581 INFO:teuthology.orchestra.run.smithi003.stdout:55834574851 2023-10-30T12:10:22.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 10 librados: watch_flush enter 2023-10-30T12:10:22.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 10 librados: watch_flush exit 2023-10-30T12:10:22.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 20 client.4342.objecter shutdown clearing up homeless session... 2023-10-30T12:10:22.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 10 client.4342.objecter successfully canceled tick 2023-10-30T12:10:22.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 1 -- 172.21.15.3:0/1163036885 >> v2:172.21.15.92:6800/1750155735 conn(0x7f63b006c810 msgr2=0x7f63b006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 1 --2- 172.21.15.3:0/1163036885 >> v2:172.21.15.92:6800/1750155735 conn(0x7f63b006c810 0x7f63b006ecd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009a40 tx=0x7f63c80091c0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 10 monclient: shutdown 2023-10-30T12:10:22.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 1 -- 172.21.15.3:0/1163036885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63c40c8310 msgr2=0x7f63c40c1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 1 --2- 172.21.15.3:0/1163036885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63c40c8310 0x7f63c40c1590 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f63c40378d0 tx=0x7f63bc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 1 -- 172.21.15.3:0/1163036885 shutdown_connections 2023-10-30T12:10:22.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 1 --2- 172.21.15.3:0/1163036885 >> v2:172.21.15.92:6800/1750155735 conn(0x7f63b006c810 0x7f63b006ecd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 1 --2- 172.21.15.3:0/1163036885 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c40c1ad0 0x7f63c4160300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 1 --2- 172.21.15.3:0/1163036885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63c40c8310 0x7f63c40c1590 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 1 --2- 172.21.15.3:0/1163036885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c4038ef0 0x7f63c402df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 1 -- 172.21.15.3:0/1163036885 >> 172.21.15.3:0/1163036885 conn(0x7f63c4006180 msgr2=0x7f63c4034630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 1 -- 172.21.15.3:0/1163036885 shutdown_connections 2023-10-30T12:10:22.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 1 -- 172.21.15.3:0/1163036885 wait complete. 2023-10-30T12:10:22.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.582+0000 7f63d4026640 1 librados: shutdown 2023-10-30T12:10:22.599 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574851 for osd.4 2023-10-30T12:10:22.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.767+0000 7fbbfc997640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T12:10:22.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.767+0000 7fbbfc997640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T12:10:22.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.767+0000 7fbbfc997640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:22.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.767+0000 7fbbfc997640 1 -- 172.21.15.3:0/4200346692 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fbbf80a8920 con 0x7fbbf80ae540 2023-10-30T12:10:22.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.782+0000 7fbbe6ffd640 1 -- 172.21.15.3:0/4200346692 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbbf007eeb0 con 0x7fbbf80ae540 2023-10-30T12:10:22.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.782+0000 7fbbe6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T12:10:22.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.782+0000 7fbbe6ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:22.784 INFO:teuthology.orchestra.run.smithi003.stdout:60129542150 2023-10-30T12:10:22.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 10 librados: watch_flush enter 2023-10-30T12:10:22.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 10 librados: watch_flush exit 2023-10-30T12:10:22.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 20 client.4362.objecter shutdown clearing up homeless session... 2023-10-30T12:10:22.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 10 client.4362.objecter successfully canceled tick 2023-10-30T12:10:22.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 1 -- 172.21.15.3:0/4200346692 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbbd806c810 msgr2=0x7fbbd806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 1 --2- 172.21.15.3:0/4200346692 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbbd806c810 0x7fbbd806ecd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80015f0 tx=0x7fbbe8007a40 comp rx=0 tx=0).stop 2023-10-30T12:10:22.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 10 monclient: shutdown 2023-10-30T12:10:22.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 1 -- 172.21.15.3:0/4200346692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbf80ae540 msgr2=0x7fbbf80af2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 1 --2- 172.21.15.3:0/4200346692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbf80ae540 0x7fbbf80af2e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0033000 tx=0x7fbbf0004a40 comp rx=0 tx=0).stop 2023-10-30T12:10:22.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 1 -- 172.21.15.3:0/4200346692 shutdown_connections 2023-10-30T12:10:22.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 1 --2- 172.21.15.3:0/4200346692 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbbd806c810 0x7fbbd806ecd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 1 --2- 172.21.15.3:0/4200346692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbf80ae540 0x7fbbf80af2e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 1 --2- 172.21.15.3:0/4200346692 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf81423f0 0x7fbbf81d7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 1 --2- 172.21.15.3:0/4200346692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf80a8e40 0x7fbbf81ca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 1 -- 172.21.15.3:0/4200346692 >> 172.21.15.3:0/4200346692 conn(0x7fbbf80b7180 msgr2=0x7fbbf80b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 1 -- 172.21.15.3:0/4200346692 shutdown_connections 2023-10-30T12:10:22.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 1 -- 172.21.15.3:0/4200346692 wait complete. 2023-10-30T12:10:22.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.784+0000 7fbbfc997640 1 librados: shutdown 2023-10-30T12:10:22.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T12:10:22.797 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.1 2023-10-30T12:10:22.797 DEBUG:teuthology.parallel:result is None 2023-10-30T12:10:22.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.897+0000 7fc5848cf640 10 monclient: get_monmap_and_config 2023-10-30T12:10:22.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.897+0000 7fc5848cf640 10 monclient: build_initial_monmap 2023-10-30T12:10:22.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.897+0000 7fc5848cf640 10 monclient: monmap: 2023-10-30T12:10:22.899 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:22.899 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:22.900 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:22.898358+0000 2023-10-30T12:10:22.900 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:22.898358+0000 2023-10-30T12:10:22.900 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:22.900 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:22.900 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:22.901 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:22.901 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:22.901 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:22.901 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:22.901 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:22.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.897+0000 7fc5848cf640 1 Processor -- start 2023-10-30T12:10:22.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 1 -- start start 2023-10-30T12:10:22.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 10 monclient: init 2023-10-30T12:10:22.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:22.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-10-30T12:10:22.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc5800a8e40 0x7fc58013efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 10 monclient(hunting): picked mon.noname-a con 0x7fc5800a8e40 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:22.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc5800b13c0 0x7fc580141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 10 monclient(hunting): picked mon.noname-c con 0x7fc5800b13c0 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:22.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc58013f500 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:22.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5800b1b00 con 0x7fc58013f500 2023-10-30T12:10:22.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5800679a0 con 0x7fc5800a8e40 2023-10-30T12:10:22.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5800b1060 con 0x7fc5800b13c0 2023-10-30T12:10:22.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:22.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.898+0000 7fc5848cf640 10 monclient(hunting): authenticate will time out at 1414.112835s 2023-10-30T12:10:22.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.899+0000 7fc57e59c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc5800a8e40 0x7fc58013efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.899+0000 7fc57dd9b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc5800b13c0 0x7fc580141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.899+0000 7fc57e59c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc5800a8e40 0x7fc58013efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38050/0 (socket says 172.21.15.3:38050) 2023-10-30T12:10:22.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.899+0000 7fc57e59c640 1 -- 172.21.15.3:0/1753353731 learned_addr learned my addr 172.21.15.3:0/1753353731 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:22.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.899+0000 7fc57e59c640 10 monclient(hunting): get_auth_request con 0x7fc5800a8e40 auth_method 0 2023-10-30T12:10:22.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.899+0000 7fc57e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.899+0000 7fc57e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.899+0000 7fc57e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.899+0000 7fc57dd9b640 10 monclient(hunting): get_auth_request con 0x7fc5800b13c0 auth_method 0 2023-10-30T12:10:22.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.899+0000 7fc57dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.899+0000 7fc57dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.899+0000 7fc57dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.900+0000 7fc57e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.900+0000 7fc57e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.900+0000 7fc57e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.900+0000 7fc57dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.900+0000 7fc57dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.900+0000 7fc57dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.902+0000 7fc57dd9b640 10 monclient(hunting): handle_auth_done global_id 4348 payload 274 2023-10-30T12:10:22.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.902+0000 7fc57dd9b640 1 -- 172.21.15.3:0/1753353731 >> v2:172.21.15.3:3300/0 conn(0x7fc5800a8e40 msgr2=0x7fc58013efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:22.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.902+0000 7fc57dd9b640 1 --2- 172.21.15.3:0/1753353731 >> v2:172.21.15.3:3300/0 conn(0x7fc5800a8e40 0x7fc58013efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.902+0000 7fc57dd9b640 1 -- 172.21.15.3:0/1753353731 >> v1:172.21.15.3:6789/0 conn(0x7fc58013f500 legacy=0x7fc5801423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.902+0000 7fc57dd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:10:22.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.902+0000 7fc57dd9b640 1 monclient: found mon.noname-c 2023-10-30T12:10:22.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.902+0000 7fc57dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:22.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.902+0000 7fc57dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:10:22.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.902+0000 7fc57dd9b640 1 -- 172.21.15.3:0/1753353731 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5800a8920 con 0x7fc5800b13c0 2023-10-30T12:10:22.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.903+0000 7fc57dd9b640 1 --2- 172.21.15.3:0/1753353731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5800b13c0 0x7fc580141eb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc580067880 tx=0x7fc56802b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91a28549c331c652 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc57d59a640 1 -- 172.21.15.3:0/1753353731 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56802f050 con 0x7fc5800b13c0 2023-10-30T12:10:22.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc57d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc57d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:22.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc57d59a640 10 monclient: dump: 2023-10-30T12:10:22.917 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:22.917 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:22.918 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.918 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.918 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.918 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:22.918 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:22.919 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:22.919 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:22.919 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:22.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc57d59a640 10 monclient: _finish_auth 0 2023-10-30T12:10:22.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc57d59a640 10 monclient: _check_auth_tickets 2023-10-30T12:10:22.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc57d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:22.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc57d59a640 1 -- 172.21.15.3:0/1753353731 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc568031710 con 0x7fc5800b13c0 2023-10-30T12:10:22.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc57d59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:22.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc57d59a640 1 -- 172.21.15.3:0/1753353731 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568031ab0 con 0x7fc5800b13c0 2023-10-30T12:10:22.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc5848cf640 5 monclient: authenticate success, global_id 4348 2023-10-30T12:10:22.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc5848cf640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:22.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc5848cf640 4 set_mon_vals no callback set 2023-10-30T12:10:22.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc57d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc57d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:22.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.907+0000 7fc57d59a640 10 monclient: dump: 2023-10-30T12:10:22.921 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:22.922 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:22.922 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.922 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.922 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.922 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:22.923 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:22.923 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:22.923 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:22.923 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:22.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.908+0000 7fc5848cf640 10 monclient: shutdown 2023-10-30T12:10:22.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.908+0000 7fc5848cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.908+0000 7fc5848cf640 1 -- 172.21.15.3:0/1753353731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5800b13c0 msgr2=0x7fc580141eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.908+0000 7fc5848cf640 1 --2- 172.21.15.3:0/1753353731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5800b13c0 0x7fc580141eb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc580067880 tx=0x7fc56802b7b0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.908+0000 7fc5848cf640 1 -- 172.21.15.3:0/1753353731 shutdown_connections 2023-10-30T12:10:22.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.908+0000 7fc5848cf640 1 --2- 172.21.15.3:0/1753353731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5800b13c0 0x7fc580141eb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.908+0000 7fc5848cf640 1 --2- 172.21.15.3:0/1753353731 >> v2:172.21.15.3:3300/0 conn(0x7fc5800a8e40 0x7fc58013efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.908+0000 7fc5848cf640 1 -- 172.21.15.3:0/1753353731 >> 172.21.15.3:0/1753353731 conn(0x7fc5800b7180 msgr2=0x7fc5800b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.908+0000 7fc5848cf640 1 -- 172.21.15.3:0/1753353731 shutdown_connections 2023-10-30T12:10:22.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.908+0000 7fc5848cf640 1 -- 172.21.15.3:0/1753353731 wait complete. 2023-10-30T12:10:22.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.909+0000 7fc5848cf640 10 monclient: build_initial_monmap 2023-10-30T12:10:22.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.909+0000 7fc5848cf640 10 monclient: monmap: 2023-10-30T12:10:22.928 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:22.928 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:22.928 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:22.929 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:22.929 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:22.929 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:22.929 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:22.929 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:22.929 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:22.930 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:22.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.909+0000 7fc5848cf640 1 librados: starting msgr at 2023-10-30T12:10:22.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.909+0000 7fc5848cf640 1 librados: starting objecter 2023-10-30T12:10:22.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.909+0000 7fc5848cf640 1 Processor -- start 2023-10-30T12:10:22.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.909+0000 7fc5848cf640 1 -- start start 2023-10-30T12:10:22.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.909+0000 7fc5848cf640 1 librados: setting wanted keys 2023-10-30T12:10:22.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.909+0000 7fc5848cf640 1 librados: calling monclient init 2023-10-30T12:10:22.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.909+0000 7fc5848cf640 10 monclient: init 2023-10-30T12:10:22.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.909+0000 7fc5848cf640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:22.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.909+0000 7fc5848cf640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:10:22.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc5848cf640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5800a8e40 0x7fc58013a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc5848cf640 10 monclient(hunting): picked mon.noname-a con 0x7fc5800a8e40 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:22.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc5848cf640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5800b2920 0x7fc5801da560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc5848cf640 10 monclient(hunting): picked mon.noname-b con 0x7fc5800b2920 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:22.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc5848cf640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc5800ad750 0x7fc580143e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc5848cf640 10 monclient(hunting): picked mon.noname-c con 0x7fc5800ad750 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:22.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc5848cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc5848cf640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5800b7ea0 con 0x7fc5800a8e40 2023-10-30T12:10:22.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc5848cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc5848cf640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5801daaa0 con 0x7fc5800b2920 2023-10-30T12:10:22.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc5848cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc5848cf640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5800ad270 con 0x7fc5800ad750 2023-10-30T12:10:22.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc5848cf640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:22.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc5848cf640 10 monclient(hunting): authenticate will time out at 1414.124118s 2023-10-30T12:10:22.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc57ed9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc5800ad750 0x7fc580143e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc57e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5800a8e40 0x7fc58013a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc57dd9b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5800b2920 0x7fc5801da560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc57dd9b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5800b2920 0x7fc5801da560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:33990/0 (socket says 172.21.15.3:33990) 2023-10-30T12:10:22.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc57dd9b640 1 -- 172.21.15.3:0/2696475202 learned_addr learned my addr 172.21.15.3:0/2696475202 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:22.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc57dd9b640 10 monclient(hunting): get_auth_request con 0x7fc5800b2920 auth_method 0 2023-10-30T12:10:22.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc57dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc57dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.910+0000 7fc57dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57ed9d640 10 monclient(hunting): get_auth_request con 0x7fc5800ad750 auth_method 0 2023-10-30T12:10:22.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57e59c640 10 monclient(hunting): get_auth_request con 0x7fc5800a8e40 auth_method 0 2023-10-30T12:10:22.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.911+0000 7fc57ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.914+0000 7fc57dd9b640 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2023-10-30T12:10:22.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.914+0000 7fc57dd9b640 1 -- 172.21.15.3:0/2696475202 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc5800ad750 msgr2=0x7fc580143e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.914+0000 7fc57dd9b640 1 --2- 172.21.15.3:0/2696475202 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc5800ad750 0x7fc580143e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.914+0000 7fc57dd9b640 1 -- 172.21.15.3:0/2696475202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5800a8e40 msgr2=0x7fc58013a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.914+0000 7fc57dd9b640 1 --2- 172.21.15.3:0/2696475202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5800a8e40 0x7fc58013a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.914+0000 7fc57dd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:10:22.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.914+0000 7fc57dd9b640 1 monclient: found mon.noname-b 2023-10-30T12:10:22.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.914+0000 7fc57dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:22.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.914+0000 7fc57dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:22.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.914+0000 7fc57dd9b640 1 -- 172.21.15.3:0/2696475202 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5801450a0 con 0x7fc5800b2920 2023-10-30T12:10:22.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T12:10:22.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.916+0000 7fc57dd9b640 1 --2- 172.21.15.3:0/2696475202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5800b2920 0x7fc5801da560 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc568009890 tx=0x7fc5680046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.916+0000 7fc576ffd640 10 client.?.objecter ms_handle_connect 0x7fc5800b2920 2023-10-30T12:10:22.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.916+0000 7fc576ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:22.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc576ffd640 1 -- 172.21.15.3:0/2696475202 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568033a40 con 0x7fc5800b2920 2023-10-30T12:10:22.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc576ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc576ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:22.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc576ffd640 10 monclient: dump: 2023-10-30T12:10:22.946 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:22.946 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:22.946 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.946 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.947 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.947 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:22.947 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:22.947 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:22.947 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:22.947 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:22.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc576ffd640 10 monclient: _finish_auth 0 2023-10-30T12:10:22.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc576ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:22.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc576ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:22.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc576ffd640 1 -- 172.21.15.3:0/2696475202 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc568033bb0 con 0x7fc5800b2920 2023-10-30T12:10:22.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc5848cf640 5 monclient: authenticate success, global_id 4351 2023-10-30T12:10:22.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc5848cf640 10 monclient: _renew_subs 2023-10-30T12:10:22.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc5848cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:22.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc5848cf640 1 -- 172.21.15.3:0/2696475202 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5800adbd0 con 0x7fc5800b2920 2023-10-30T12:10:22.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc576ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:22.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc576ffd640 1 -- 172.21.15.3:0/2696475202 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56803c400 con 0x7fc5800b2920 2023-10-30T12:10:22.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc576ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc576ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:22.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc576ffd640 10 monclient: dump: 2023-10-30T12:10:22.950 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:22.951 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:22.951 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.951 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:22.951 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.951 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:22.952 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:22.952 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:22.952 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:22.952 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:22.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc57cd99640 4 set_mon_vals no callback set 2023-10-30T12:10:22.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc5848cf640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:22.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc5848cf640 10 monclient: _renew_subs 2023-10-30T12:10:22.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc5848cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:22.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc5848cf640 1 -- 172.21.15.3:0/2696475202 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc580145cc0 con 0x7fc5800b2920 2023-10-30T12:10:22.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.919+0000 7fc5848cf640 1 librados: init done 2023-10-30T12:10:22.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.920+0000 7fc5848cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.920+0000 7fc5848cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.920+0000 7fc5848cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:22.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.920+0000 7fc5848cf640 1 -- 172.21.15.3:0/2696475202 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc580145e60 con 0x7fc5800b2920 2023-10-30T12:10:22.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.939+0000 7fc576ffd640 1 -- 172.21.15.3:0/2696475202 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fc56803a040 con 0x7fc5800b2920 2023-10-30T12:10:22.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.940+0000 7fc576ffd640 1 --2- 172.21.15.3:0/2696475202 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc56006c7c0 0x7fc56006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.940+0000 7fc57e59c640 1 --2- 172.21.15.3:0/2696475202 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc56006c7c0 0x7fc56006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.940+0000 7fc57e59c640 10 monclient: get_auth_request con 0x7fc56006c7c0 auth_method 0 2023-10-30T12:10:22.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.940+0000 7fc57e59c640 1 --2- 172.21.15.3:0/2696475202 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc56006c7c0 0x7fc56006ec80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc578009bd0 tx=0x7fc578009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:22.940+0000 7fc576ffd640 10 client.4351.objecter ms_handle_connect 0x7fc56006c7c0 2023-10-30T12:10:23.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.060+0000 7f38f9e1f640 10 monclient: get_monmap_and_config 2023-10-30T12:10:23.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.060+0000 7f38f9e1f640 10 monclient: build_initial_monmap 2023-10-30T12:10:23.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.060+0000 7f38f9e1f640 10 monclient: monmap: 2023-10-30T12:10:23.062 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:23.062 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.063 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:23.061356+0000 2023-10-30T12:10:23.063 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:23.061356+0000 2023-10-30T12:10:23.063 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:23.063 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.063 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:23.063 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:23.064 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:23.064 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:23.064 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:23.064 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.061+0000 7f38f9e1f640 1 Processor -- start 2023-10-30T12:10:23.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.061+0000 7f38f9e1f640 1 -- start start 2023-10-30T12:10:23.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.061+0000 7f38f9e1f640 10 monclient: init 2023-10-30T12:10:23.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.061+0000 7f38f9e1f640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:23.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.061+0000 7f38f9e1f640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-10-30T12:10:23.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.061+0000 7f38f9e1f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38f406be50 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:23.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.061+0000 7f38f9e1f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f38f4071a90 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:23.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.061+0000 7f38f9e1f640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f38f4145970 0x7f38f406b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.061+0000 7f38f9e1f640 10 monclient(hunting): picked mon.noname-c con 0x7f38f4145970 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:23.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.061+0000 7f38f9e1f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f4144dd0 con 0x7f38f4071a90 2023-10-30T12:10:23.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.061+0000 7f38f9e1f640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f40b11d0 con 0x7f38f406be50 2023-10-30T12:10:23.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.061+0000 7f38f9e1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.061+0000 7f38f9e1f640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38f40679f0 con 0x7f38f4145970 2023-10-30T12:10:23.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.061+0000 7f38f9e1f640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:23.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.061+0000 7f38f9e1f640 10 monclient(hunting): authenticate will time out at 1414.275851s 2023-10-30T12:10:23.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.062+0000 7f38f2ffd640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f38f4071a90 0x7f38f4145260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:39582/0 (socket says 172.21.15.3:39582) 2023-10-30T12:10:23.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.062+0000 7f38f2ffd640 1 -- 172.21.15.3:0/3662540109 learned_addr learned my addr 172.21.15.3:0/3662540109 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:23.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.064+0000 7f38f27fc640 1 -- 172.21.15.3:0/3662540109 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159095680 0 0) 0x7f38e8001110 con 0x7f38f4071a90 2023-10-30T12:10:23.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.064+0000 7f38f27fc640 10 monclient(hunting): _negotiate 2023-10-30T12:10:23.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.064+0000 7f38f27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.064+0000 7f38f27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.064+0000 7f38f27fc640 10 monclient(hunting): my global_id is 4371 2023-10-30T12:10:23.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.064+0000 7f38f27fc640 1 -- 172.21.15.3:0/3662540109 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38dc003520 con 0x7f38f4071a90 2023-10-30T12:10:23.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 1 -- 172.21.15.3:0/3662540109 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 106379642 0 0) 0x7f38e8001510 con 0x7f38f4071a90 2023-10-30T12:10:23.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 1 -- 172.21.15.3:0/3662540109 >> v2:172.21.15.92:3300/0 conn(0x7f38f4145970 msgr2=0x7f38f406b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:23.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 1 --2- 172.21.15.3:0/3662540109 >> v2:172.21.15.92:3300/0 conn(0x7f38f4145970 0x7f38f406b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 1 -- 172.21.15.3:0/3662540109 >> v1:172.21.15.92:6789/0 conn(0x7f38f406be50 legacy=0x7f38f4071380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:23.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient: _finish_hunting 0 2023-10-30T12:10:23.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:10:23.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:23.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.3:6789/0 2023-10-30T12:10:23.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 1 -- 172.21.15.3:0/3662540109 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f40af8f0 con 0x7f38f4071a90 2023-10-30T12:10:23.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient: _finish_auth 0 2023-10-30T12:10:23.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:10:23.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:23.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 1 -- 172.21.15.3:0/3662540109 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1777801622 0 0) 0x7f38e80019b0 con 0x7f38f4071a90 2023-10-30T12:10:23.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:10:23.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient: dump: 2023-10-30T12:10:23.074 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.074 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.074 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.074 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.075 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.075 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.075 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.075 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.075 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.076 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] but i'm connected to v1:172.21.15.3:6789/0, reconnecting 2023-10-30T12:10:23.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:23.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 1 -- 172.21.15.3:0/3662540109 >> v1:172.21.15.3:6789/0 conn(0x7f38f4071a90 legacy=0x7f38f4145260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:23.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 1 --2- 172.21.15.3:0/3662540109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38dc0039c0 0x7f38dc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient(hunting): picked mon.b con 0x7f38dc0039c0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:23.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 1 --2- 172.21.15.3:0/3662540109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38dc0047c0 0x7f38dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient(hunting): picked mon.a con 0x7f38dc0047c0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:23.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 1 --2- 172.21.15.3:0/3662540109 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38dc005330 0x7f38dc00b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient(hunting): picked mon.c con 0x7f38dc005330 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:23.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 1 -- 172.21.15.3:0/3662540109 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f40679f0 con 0x7f38dc0047c0 2023-10-30T12:10:23.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 1 -- 172.21.15.3:0/3662540109 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc00c280 con 0x7f38dc0039c0 2023-10-30T12:10:23.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 1 -- 172.21.15.3:0/3662540109 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc00c450 con 0x7f38dc005330 2023-10-30T12:10:23.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:23.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.068+0000 7f38f27fc640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:10:23.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.069+0000 7f38f3fff640 1 --2- 172.21.15.3:0/3662540109 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38dc005330 0x7f38dc00b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.069+0000 7f38f3fff640 10 monclient(hunting): get_auth_request con 0x7f38dc005330 auth_method 0 2023-10-30T12:10:23.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.069+0000 7f38f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.069+0000 7f38f3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.069+0000 7f38f3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:23.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.069+0000 7f38f2ffd640 1 --2- 172.21.15.3:0/3662540109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38dc0047c0 0x7f38dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.070+0000 7f38f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.070+0000 7f38f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.070+0000 7f38f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:10:23.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f2ffd640 10 monclient(hunting): get_auth_request con 0x7f38dc0047c0 auth_method 0 2023-10-30T12:10:23.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:23.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f3fff640 10 monclient(hunting): handle_auth_done global_id 4371 payload 293 2023-10-30T12:10:23.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f3fff640 1 -- 172.21.15.3:0/3662540109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38dc0039c0 msgr2=0x7f38dc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:23.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f3fff640 1 --2- 172.21.15.3:0/3662540109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38dc0039c0 0x7f38dc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f3fff640 1 -- 172.21.15.3:0/3662540109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38dc0047c0 msgr2=0x7f38dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:23.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f3fff640 1 --2- 172.21.15.3:0/3662540109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38dc0047c0 0x7f38dc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f3fff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:23.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f3fff640 1 monclient: found mon.c 2023-10-30T12:10:23.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:23.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f3fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f3fff640 1 -- 172.21.15.3:0/3662540109 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e8001510 con 0x7f38dc005330 2023-10-30T12:10:23.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f3fff640 10 monclient: _finish_auth 0 2023-10-30T12:10:23.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:23.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:23.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f9e1f640 5 monclient: authenticate success, global_id 4371 2023-10-30T12:10:23.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.072+0000 7f38f9e1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:23.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.073+0000 7f38f3fff640 1 -- 172.21.15.3:0/3662540109 reap_dead start 2023-10-30T12:10:23.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.074+0000 7f38f3fff640 1 --2- 172.21.15.3:0/3662540109 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38dc005330 0x7f38dc00b9a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f38ec009e40 tx=0x7f38dc0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cefa12474ba743f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f27fc640 1 -- 172.21.15.3:0/3662540109 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec02cd90 con 0x7f38dc005330 2023-10-30T12:10:23.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:23.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f27fc640 10 monclient: dump: 2023-10-30T12:10:23.089 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.089 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.090 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.090 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.090 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.090 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.090 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.091 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.091 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.091 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f27fc640 1 -- 172.21.15.3:0/3662540109 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ec003990 con 0x7f38dc005330 2023-10-30T12:10:23.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f9e1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:23.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:23.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f9e1f640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:23.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f27fc640 1 -- 172.21.15.3:0/3662540109 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec003d50 con 0x7f38dc005330 2023-10-30T12:10:23.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f9e1f640 4 set_mon_vals no callback set 2023-10-30T12:10:23.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:23.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f27fc640 10 monclient: dump: 2023-10-30T12:10:23.093 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.093 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.093 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.093 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.094 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.094 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.094 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.094 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.094 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.094 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f9e1f640 10 monclient: shutdown 2023-10-30T12:10:23.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f9e1f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:23.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f9e1f640 1 -- 172.21.15.3:0/3662540109 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38dc005330 msgr2=0x7f38dc00b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f9e1f640 1 --2- 172.21.15.3:0/3662540109 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38dc005330 0x7f38dc00b9a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f38ec009e40 tx=0x7f38dc0057f0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f9e1f640 1 -- 172.21.15.3:0/3662540109 shutdown_connections 2023-10-30T12:10:23.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f9e1f640 1 --2- 172.21.15.3:0/3662540109 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38dc005330 0x7f38dc00b9a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.078+0000 7f38f9e1f640 1 -- 172.21.15.3:0/3662540109 >> 172.21.15.3:0/3662540109 conn(0x7f38f4066470 msgr2=0x7f38f40b4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:23.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.079+0000 7f38f9e1f640 1 -- 172.21.15.3:0/3662540109 shutdown_connections 2023-10-30T12:10:23.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.079+0000 7f38f9e1f640 1 -- 172.21.15.3:0/3662540109 wait complete. 2023-10-30T12:10:23.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.079+0000 7f38f9e1f640 10 monclient: build_initial_monmap 2023-10-30T12:10:23.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.079+0000 7f38f9e1f640 10 monclient: monmap: 2023-10-30T12:10:23.097 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:23.097 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:23.097 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:23.097 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:23.098 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:23.098 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:23.098 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:23.098 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:23.098 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:23.098 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.080+0000 7f38f9e1f640 1 librados: starting msgr at 2023-10-30T12:10:23.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.080+0000 7f38f9e1f640 1 librados: starting objecter 2023-10-30T12:10:23.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.080+0000 7f38f9e1f640 1 Processor -- start 2023-10-30T12:10:23.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.080+0000 7f38f9e1f640 1 -- start start 2023-10-30T12:10:23.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.080+0000 7f38f9e1f640 1 librados: setting wanted keys 2023-10-30T12:10:23.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.080+0000 7f38f9e1f640 1 librados: calling monclient init 2023-10-30T12:10:23.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.080+0000 7f38f9e1f640 10 monclient: init 2023-10-30T12:10:23.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.080+0000 7f38f9e1f640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:23.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.080+0000 7f38f9e1f640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:10:23.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.080+0000 7f38f9e1f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38dc005330 0x7f38f40ad100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.080+0000 7f38f9e1f640 10 monclient(hunting): picked mon.noname-a con 0x7f38dc005330 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:23.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.080+0000 7f38f9e1f640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38f40ad640 0x7f38f41da220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.080+0000 7f38f9e1f640 10 monclient(hunting): picked mon.noname-b con 0x7f38f40ad640 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:23.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f9e1f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38f41da760 0x7f38f41d66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f9e1f640 10 monclient(hunting): picked mon.noname-c con 0x7f38f41da760 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:23.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f9e1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f9e1f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f4142910 con 0x7f38dc005330 2023-10-30T12:10:23.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f9e1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f9e1f640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f41d9d90 con 0x7f38f40ad640 2023-10-30T12:10:23.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f9e1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f9e1f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f4142360 con 0x7f38f41da760 2023-10-30T12:10:23.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f9e1f640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:23.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f9e1f640 10 monclient(hunting): authenticate will time out at 1414.295101s 2023-10-30T12:10:23.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38dc005330 0x7f38f40ad100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f3fff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38f41da760 0x7f38f41d66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f3fff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38f41da760 0x7f38f41d66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:48428/0 (socket says 172.21.15.3:48428) 2023-10-30T12:10:23.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38dc005330 0x7f38f40ad100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38098/0 (socket says 172.21.15.3:38098) 2023-10-30T12:10:23.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f3fff640 1 -- 172.21.15.3:0/1721248034 learned_addr learned my addr 172.21.15.3:0/1721248034 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:23.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f3fff640 10 monclient(hunting): get_auth_request con 0x7f38f41da760 auth_method 0 2023-10-30T12:10:23.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f37fe640 10 monclient(hunting): get_auth_request con 0x7f38dc005330 auth_method 0 2023-10-30T12:10:23.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.081+0000 7f38f37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T12:10:23.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.082+0000 7f38f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.082+0000 7f38f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.082+0000 7f38f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.082+0000 7f38f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.082+0000 7f38f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.082+0000 7f38f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.085+0000 7f38f3fff640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2023-10-30T12:10:23.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.085+0000 7f38f3fff640 1 -- 172.21.15.3:0/1721248034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38f40ad640 msgr2=0x7f38f41da220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.085+0000 7f38f3fff640 1 --2- 172.21.15.3:0/1721248034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38f40ad640 0x7f38f41da220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.085+0000 7f38f3fff640 1 -- 172.21.15.3:0/1721248034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38dc005330 msgr2=0x7f38f40ad100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.085+0000 7f38f3fff640 1 --2- 172.21.15.3:0/1721248034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38dc005330 0x7f38f40ad100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.085+0000 7f38f3fff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:23.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.085+0000 7f38f3fff640 1 monclient: found mon.noname-c 2023-10-30T12:10:23.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.085+0000 7f38f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:23.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.085+0000 7f38f3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.085+0000 7f38f3fff640 1 -- 172.21.15.3:0/1721248034 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f4140620 con 0x7f38f41da760 2023-10-30T12:10:23.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.087+0000 7f38f3fff640 1 --2- 172.21.15.3:0/1721248034 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38f41da760 0x7f38f41d66b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f38ec035680 tx=0x7f38ec004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.087+0000 7f38dbfff640 10 client.?.objecter ms_handle_connect 0x7f38f41da760 2023-10-30T12:10:23.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.087+0000 7f38dbfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:23.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38dbfff640 1 -- 172.21.15.3:0/1721248034 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec035b30 con 0x7f38f41da760 2023-10-30T12:10:23.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38dbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:23.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38dbfff640 10 monclient: dump: 2023-10-30T12:10:23.112 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.113 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.113 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.113 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.113 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.113 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.113 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.114 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.114 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.114 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38dbfff640 10 monclient: _finish_auth 0 2023-10-30T12:10:23.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38dbfff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:23.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:23.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38dbfff640 1 -- 172.21.15.3:0/1721248034 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ec035ca0 con 0x7f38f41da760 2023-10-30T12:10:23.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38dbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:23.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38dbfff640 1 -- 172.21.15.3:0/1721248034 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec03d600 con 0x7f38f41da760 2023-10-30T12:10:23.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:23.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38dbfff640 10 monclient: dump: 2023-10-30T12:10:23.116 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.116 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.116 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.118 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.118 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38f9e1f640 5 monclient: authenticate success, global_id 4289 2023-10-30T12:10:23.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38f9e1f640 10 monclient: _renew_subs 2023-10-30T12:10:23.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38f9e1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38f9e1f640 1 -- 172.21.15.3:0/1721248034 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f40adb70 con 0x7f38f41da760 2023-10-30T12:10:23.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38f1ffb640 4 set_mon_vals no callback set 2023-10-30T12:10:23.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38f9e1f640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:23.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38f9e1f640 10 monclient: _renew_subs 2023-10-30T12:10:23.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38f9e1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38f9e1f640 1 -- 172.21.15.3:0/1721248034 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f41d6dd0 con 0x7f38f41da760 2023-10-30T12:10:23.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.091+0000 7f38f9e1f640 1 librados: init done 2023-10-30T12:10:23.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.092+0000 7f38f9e1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.092+0000 7f38f9e1f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.092+0000 7f38f9e1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.092+0000 7f38f9e1f640 1 -- 172.21.15.3:0/1721248034 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f41d9bf0 con 0x7f38f41da760 2023-10-30T12:10:23.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.164+0000 7fc576ffd640 1 -- 172.21.15.3:0/2696475202 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fc568079490 con 0x7fc5800b2920 2023-10-30T12:10:23.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.164+0000 7fc576ffd640 10 client.4351.objecter ms_dispatch 0x7fc580000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:23.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.164+0000 7fc576ffd640 3 client.4351.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:23.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.164+0000 7fc576ffd640 3 client.4351.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:23.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.164+0000 7fc576ffd640 20 client.4351.objecter dump_active .. 0 homeless 2023-10-30T12:10:23.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.173+0000 7fc576ffd640 1 -- 172.21.15.3:0/2696475202 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc56807d530 con 0x7fc5800b2920 2023-10-30T12:10:23.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.174+0000 7fc576ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.174+0000 7fc576ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:23.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.226+0000 7fc082234640 10 monclient: get_monmap_and_config 2023-10-30T12:10:23.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.226+0000 7fc082234640 10 monclient: build_initial_monmap 2023-10-30T12:10:23.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.226+0000 7fc082234640 10 monclient: monmap: 2023-10-30T12:10:23.228 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:23.228 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.229 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:23.227595+0000 2023-10-30T12:10:23.229 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:23.227595+0000 2023-10-30T12:10:23.229 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:23.229 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.229 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:23.230 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:23.230 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:23.230 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:23.230 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:23.231 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.227+0000 7fc082234640 1 Processor -- start 2023-10-30T12:10:23.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.227+0000 7fc082234640 1 -- start start 2023-10-30T12:10:23.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.227+0000 7fc082234640 10 monclient: init 2023-10-30T12:10:23.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.227+0000 7fc082234640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:23.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.227+0000 7fc082234640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-10-30T12:10:23.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.227+0000 7fc082234640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc07c0b4350 0x7fc07c0a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.228+0000 7fc082234640 10 monclient(hunting): picked mon.noname-b con 0x7fc07c0b4350 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:23.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.228+0000 7fc082234640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc07c13eb50 0x7fc07c13ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.228+0000 7fc082234640 10 monclient(hunting): picked mon.noname-c con 0x7fc07c13eb50 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:23.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.228+0000 7fc082234640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc07c144f50 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:23.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.228+0000 7fc082234640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07c0aa6e0 con 0x7fc07c144f50 2023-10-30T12:10:23.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.228+0000 7fc082234640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.228+0000 7fc082234640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc07c0679f0 con 0x7fc07c13eb50 2023-10-30T12:10:23.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.228+0000 7fc082234640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.228+0000 7fc082234640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c0b4730 con 0x7fc07c0b4350 2023-10-30T12:10:23.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.228+0000 7fc082234640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:23.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.228+0000 7fc082234640 10 monclient(hunting): authenticate will time out at 1414.442133s 2023-10-30T12:10:23.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.228+0000 7fc07affd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc07c13eb50 0x7fc07c13ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.228+0000 7fc07bfff640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fc07c144f50 0x7fc07c1453e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.3:33270/0 (socket says 172.21.15.3:33270) 2023-10-30T12:10:23.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.228+0000 7fc07bfff640 1 -- 172.21.15.3:0/3094140681 learned_addr learned my addr 172.21.15.3:0/3094140681 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:23.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.229+0000 7fc07affd640 10 monclient(hunting): get_auth_request con 0x7fc07c13eb50 auth_method 0 2023-10-30T12:10:23.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.229+0000 7fc07affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.229+0000 7fc07affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.229+0000 7fc07affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.229+0000 7fc07affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.229+0000 7fc07affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.229+0000 7fc07affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.231+0000 7fc07affd640 10 monclient(hunting): handle_auth_done global_id 4354 payload 274 2023-10-30T12:10:23.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.231+0000 7fc07affd640 1 -- 172.21.15.3:0/3094140681 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc07c0b4350 msgr2=0x7fc07c0a83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:23.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.231+0000 7fc07affd640 1 --2- 172.21.15.3:0/3094140681 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc07c0b4350 0x7fc07c0a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.231+0000 7fc07affd640 1 -- 172.21.15.3:0/3094140681 >> v1:172.21.15.92:6789/0 conn(0x7fc07c144f50 legacy=0x7fc07c1453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.232+0000 7fc07affd640 10 monclient: _finish_hunting 0 2023-10-30T12:10:23.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.232+0000 7fc07affd640 1 monclient: found mon.noname-c 2023-10-30T12:10:23.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.232+0000 7fc07affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:23.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.232+0000 7fc07affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:10:23.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.232+0000 7fc07affd640 1 -- 172.21.15.3:0/3094140681 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07c0a8910 con 0x7fc07c13eb50 2023-10-30T12:10:23.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.235+0000 7fc07affd640 1 --2- 172.21.15.3:0/3094140681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07c13eb50 0x7fc07c13ef70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc0700097a0 tx=0x7fc07002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4a6a28dfd5d7422 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc07a7fc640 1 -- 172.21.15.3:0/3094140681 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070031700 con 0x7fc07c13eb50 2023-10-30T12:10:23.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc07a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc07a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:23.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc07a7fc640 10 monclient: dump: 2023-10-30T12:10:23.241 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.242 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.242 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.242 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.242 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.242 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.243 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.243 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.243 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.243 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc07a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:10:23.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc07a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:10:23.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc07a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:23.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc07a7fc640 1 -- 172.21.15.3:0/3094140681 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc070031870 con 0x7fc07c13eb50 2023-10-30T12:10:23.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc082234640 5 monclient: authenticate success, global_id 4354 2023-10-30T12:10:23.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc082234640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:23.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc07a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:23.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc082234640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:23.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc07a7fc640 1 -- 172.21.15.3:0/3094140681 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070031c10 con 0x7fc07c13eb50 2023-10-30T12:10:23.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc07a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc082234640 4 set_mon_vals no callback set 2023-10-30T12:10:23.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc07a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:23.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.239+0000 7fc07a7fc640 10 monclient: dump: 2023-10-30T12:10:23.246 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.246 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.246 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.246 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.247 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.247 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.247 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.247 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.247 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.248 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.240+0000 7fc082234640 10 monclient: shutdown 2023-10-30T12:10:23.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.240+0000 7fc082234640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:23.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.240+0000 7fc082234640 1 -- 172.21.15.3:0/3094140681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07c13eb50 msgr2=0x7fc07c13ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.240+0000 7fc082234640 1 --2- 172.21.15.3:0/3094140681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07c13eb50 0x7fc07c13ef70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc0700097a0 tx=0x7fc07002b7a0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.240+0000 7fc082234640 1 -- 172.21.15.3:0/3094140681 shutdown_connections 2023-10-30T12:10:23.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.240+0000 7fc082234640 1 --2- 172.21.15.3:0/3094140681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07c13eb50 0x7fc07c13ef70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.240+0000 7fc082234640 1 --2- 172.21.15.3:0/3094140681 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc07c0b4350 0x7fc07c0a83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.240+0000 7fc082234640 1 -- 172.21.15.3:0/3094140681 >> 172.21.15.3:0/3094140681 conn(0x7fc07c065ea0 msgr2=0x7fc07c0662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:23.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.240+0000 7fc082234640 1 -- 172.21.15.3:0/3094140681 shutdown_connections 2023-10-30T12:10:23.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.240+0000 7fc082234640 1 -- 172.21.15.3:0/3094140681 wait complete. 2023-10-30T12:10:23.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.241+0000 7fc082234640 10 monclient: build_initial_monmap 2023-10-30T12:10:23.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.241+0000 7fc082234640 10 monclient: monmap: 2023-10-30T12:10:23.253 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:23.253 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:23.253 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:23.253 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:23.254 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:23.254 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:23.254 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:23.254 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:23.254 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:23.255 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.241+0000 7fc082234640 1 librados: starting msgr at 2023-10-30T12:10:23.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.241+0000 7fc082234640 1 librados: starting objecter 2023-10-30T12:10:23.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.241+0000 7fc082234640 1 Processor -- start 2023-10-30T12:10:23.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.241+0000 7fc082234640 1 -- start start 2023-10-30T12:10:23.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.241+0000 7fc082234640 1 librados: setting wanted keys 2023-10-30T12:10:23.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.241+0000 7fc082234640 1 librados: calling monclient init 2023-10-30T12:10:23.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.241+0000 7fc082234640 10 monclient: init 2023-10-30T12:10:23.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:23.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:23.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07c0b4350 0x7fc07c0b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 10 monclient(hunting): picked mon.noname-b con 0x7fc07c0b4350 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:23.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc07c13eb50 0x7fc07c0b1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 10 monclient(hunting): picked mon.noname-a con 0x7fc07c13eb50 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:23.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc07c144f50 0x7fc07c1d6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 10 monclient(hunting): picked mon.noname-c con 0x7fc07c144f50 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:23.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c13f190 con 0x7fc07c13eb50 2023-10-30T12:10:23.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c0b1540 con 0x7fc07c0b4350 2023-10-30T12:10:23.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c1427b0 con 0x7fc07c144f50 2023-10-30T12:10:23.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:23.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc082234640 10 monclient(hunting): authenticate will time out at 1414.456336s 2023-10-30T12:10:23.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.242+0000 7fc07b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07c0b4350 0x7fc07c0b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.243+0000 7fc07affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc07c13eb50 0x7fc07c0b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.243+0000 7fc07b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07c0b4350 0x7fc07c0b0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34034/0 (socket says 172.21.15.3:34034) 2023-10-30T12:10:23.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.243+0000 7fc07b7fe640 1 -- 172.21.15.3:0/3261684186 learned_addr learned my addr 172.21.15.3:0/3261684186 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:23.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.243+0000 7fc07b7fe640 10 monclient(hunting): get_auth_request con 0x7fc07c0b4350 auth_method 0 2023-10-30T12:10:23.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.243+0000 7fc07b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.243+0000 7fc07b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.243+0000 7fc07b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.243+0000 7fc07affd640 10 monclient(hunting): get_auth_request con 0x7fc07c13eb50 auth_method 0 2023-10-30T12:10:23.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.243+0000 7fc07affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.243+0000 7fc07affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.243+0000 7fc07affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.243+0000 7fc07b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.243+0000 7fc07b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.243+0000 7fc07b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.244+0000 7fc07affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.244+0000 7fc07affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.244+0000 7fc07affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.246+0000 7fc07b7fe640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2023-10-30T12:10:23.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.246+0000 7fc07b7fe640 1 -- 172.21.15.3:0/3261684186 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc07c144f50 msgr2=0x7fc07c1d6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.246+0000 7fc07b7fe640 1 --2- 172.21.15.3:0/3261684186 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc07c144f50 0x7fc07c1d6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.246+0000 7fc07b7fe640 1 -- 172.21.15.3:0/3261684186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc07c13eb50 msgr2=0x7fc07c0b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.246+0000 7fc07b7fe640 1 --2- 172.21.15.3:0/3261684186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc07c13eb50 0x7fc07c0b1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.246+0000 7fc07b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:10:23.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.246+0000 7fc07b7fe640 1 monclient: found mon.noname-b 2023-10-30T12:10:23.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.246+0000 7fc07b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:23.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.246+0000 7fc07b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:23.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.246+0000 7fc07b7fe640 1 -- 172.21.15.3:0/3261684186 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07c1d7000 con 0x7fc07c0b4350 2023-10-30T12:10:23.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.248+0000 7fc07b7fe640 1 --2- 172.21.15.3:0/3261684186 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07c0b4350 0x7fc07c0b0a30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00b4d0 tx=0x7fc06c00b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.248+0000 7fc063fff640 10 client.?.objecter ms_handle_connect 0x7fc07c0b4350 2023-10-30T12:10:23.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.248+0000 7fc063fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:23.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc063fff640 1 -- 172.21.15.3:0/3261684186 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c010870 con 0x7fc07c0b4350 2023-10-30T12:10:23.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc063fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc063fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:23.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc063fff640 10 monclient: dump: 2023-10-30T12:10:23.266 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.266 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.266 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.267 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.267 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.267 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.267 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.267 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.268 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.268 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc063fff640 10 monclient: _finish_auth 0 2023-10-30T12:10:23.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc063fff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:23.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc063fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:23.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc063fff640 1 -- 172.21.15.3:0/3261684186 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06c0109e0 con 0x7fc07c0b4350 2023-10-30T12:10:23.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc063fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:23.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc063fff640 1 -- 172.21.15.3:0/3261684186 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c010da0 con 0x7fc07c0b4350 2023-10-30T12:10:23.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc063fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc063fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:23.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc063fff640 10 monclient: dump: 2023-10-30T12:10:23.270 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.270 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.270 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.270 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.270 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.270 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.271 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.271 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.271 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.271 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc082234640 5 monclient: authenticate success, global_id 4360 2023-10-30T12:10:23.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc082234640 10 monclient: _renew_subs 2023-10-30T12:10:23.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc079ffb640 4 set_mon_vals no callback set 2023-10-30T12:10:23.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc082234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:23.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.251+0000 7fc082234640 1 -- 172.21.15.3:0/3261684186 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc07c1419d0 con 0x7fc07c0b4350 2023-10-30T12:10:23.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.252+0000 7fc082234640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:23.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.252+0000 7fc082234640 10 monclient: _renew_subs 2023-10-30T12:10:23.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.252+0000 7fc082234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:23.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.252+0000 7fc082234640 1 -- 172.21.15.3:0/3261684186 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc07c0aef10 con 0x7fc07c0b4350 2023-10-30T12:10:23.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.252+0000 7fc082234640 1 librados: init done 2023-10-30T12:10:23.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.252+0000 7fc082234640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.252+0000 7fc082234640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.252+0000 7fc082234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:23.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.252+0000 7fc082234640 1 -- 172.21.15.3:0/3261684186 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc07c142610 con 0x7fc07c0b4350 2023-10-30T12:10:23.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.271+0000 7fc063fff640 1 -- 172.21.15.3:0/3261684186 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fc06c023480 con 0x7fc07c0b4350 2023-10-30T12:10:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.271+0000 7fc063fff640 1 --2- 172.21.15.3:0/3261684186 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc06806c810 0x7fc06806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.271+0000 7fc07affd640 1 --2- 172.21.15.3:0/3261684186 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc06806c810 0x7fc06806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.271+0000 7fc07affd640 10 monclient: get_auth_request con 0x7fc06806c810 auth_method 0 2023-10-30T12:10:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.272+0000 7fc063fff640 10 client.4360.objecter ms_handle_connect 0x7fc06806c810 2023-10-30T12:10:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.272+0000 7fc07affd640 1 --2- 172.21.15.3:0/3261684186 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc06806c810 0x7fc06806ecd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc070035000 tx=0x7fc070008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.274+0000 7fc063fff640 1 -- 172.21.15.3:0/3261684186 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fc06c058150 con 0x7fc07c0b4350 2023-10-30T12:10:23.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.274+0000 7fc063fff640 10 client.4360.objecter ms_dispatch 0x7fc07c000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:23.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.274+0000 7fc063fff640 3 client.4360.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:23.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.274+0000 7fc063fff640 3 client.4360.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:23.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.274+0000 7fc063fff640 20 client.4360.objecter dump_active .. 0 homeless 2023-10-30T12:10:23.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T12:10:23.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.358+0000 7fc5848cf640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T12:10:23.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.358+0000 7fc5848cf640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T12:10:23.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.358+0000 7fc5848cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:23.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.358+0000 7fc5848cf640 1 -- 172.21.15.3:0/2696475202 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fc5801450a0 con 0x7fc5800b2920 2023-10-30T12:10:23.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.360+0000 7f38dbfff640 1 -- 172.21.15.3:0/1721248034 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f38ec03d770 con 0x7f38f41da760 2023-10-30T12:10:23.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.361+0000 7f38dbfff640 1 --2- 172.21.15.3:0/1721248034 >> v2:172.21.15.92:6800/1750155735 conn(0x7f38e006c7c0 0x7f38e006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.361+0000 7f38f37fe640 1 --2- 172.21.15.3:0/1721248034 >> v2:172.21.15.92:6800/1750155735 conn(0x7f38e006c7c0 0x7f38e006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.361+0000 7f38f37fe640 10 monclient: get_auth_request con 0x7f38e006c7c0 auth_method 0 2023-10-30T12:10:23.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.361+0000 7f38dbfff640 10 client.4289.objecter ms_handle_connect 0x7f38e006c7c0 2023-10-30T12:10:23.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.361+0000 7f38f37fe640 1 --2- 172.21.15.3:0/1721248034 >> v2:172.21.15.92:6800/1750155735 conn(0x7f38e006c7c0 0x7f38e006ec80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f38e4001780 tx=0x7f38e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.374+0000 7f38dbfff640 1 -- 172.21.15.3:0/1721248034 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f38ec035e20 con 0x7f38f41da760 2023-10-30T12:10:23.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.374+0000 7f38dbfff640 10 client.4289.objecter ms_dispatch 0x7f38f4000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:23.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.374+0000 7f38dbfff640 3 client.4289.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:23.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.374+0000 7f38dbfff640 3 client.4289.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:23.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.374+0000 7f38dbfff640 20 client.4289.objecter dump_active .. 0 homeless 2023-10-30T12:10:23.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.375+0000 7f38dbfff640 1 -- 172.21.15.3:0/1721248034 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f38ec043070 con 0x7f38f41da760 2023-10-30T12:10:23.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.375+0000 7f38dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.375+0000 7f38dbfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:23.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.392+0000 7fd0f7ba9640 10 monclient: get_monmap_and_config 2023-10-30T12:10:23.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.392+0000 7fd0f7ba9640 10 monclient: build_initial_monmap 2023-10-30T12:10:23.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.392+0000 7fd0f7ba9640 10 monclient: monmap: 2023-10-30T12:10:23.394 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:23.394 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.394 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:23.393541+0000 2023-10-30T12:10:23.395 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:23.393541+0000 2023-10-30T12:10:23.395 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:23.395 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.395 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:23.395 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:23.396 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:23.396 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:23.396 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:23.396 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.393+0000 7fd0f7ba9640 1 Processor -- start 2023-10-30T12:10:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.393+0000 7fd0f7ba9640 1 -- start start 2023-10-30T12:10:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.393+0000 7fd0f7ba9640 10 monclient: init 2023-10-30T12:10:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2023-10-30T12:10:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0f00b4350 0x7fd0f00a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 10 monclient(hunting): picked mon.noname-b con 0x7fd0f00b4350 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:23.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd0f013eb50 0x7fd0f013ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 10 monclient(hunting): picked mon.noname-a con 0x7fd0f013eb50 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:23.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd0f0144f50 0x7fd0f01453e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 10 monclient(hunting): picked mon.noname-c con 0x7fd0f0144f50 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:23.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f00679f0 con 0x7fd0f013eb50 2023-10-30T12:10:23.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f00b4730 con 0x7fd0f0144f50 2023-10-30T12:10:23.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00aa6e0 con 0x7fd0f00b4350 2023-10-30T12:10:23.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:23.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f7ba9640 10 monclient(hunting): authenticate will time out at 1414.608190s 2023-10-30T12:10:23.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f5945640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0f00b4350 0x7fd0f00a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.394+0000 7fd0f5144640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd0f013eb50 0x7fd0f013ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5945640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0f00b4350 0x7fd0f00a83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:48458/0 (socket says 172.21.15.3:48458) 2023-10-30T12:10:23.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5144640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd0f013eb50 0x7fd0f013ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38112/0 (socket says 172.21.15.3:38112) 2023-10-30T12:10:23.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5945640 1 -- 172.21.15.3:0/3994605105 learned_addr learned my addr 172.21.15.3:0/3994605105 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:23.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5945640 10 monclient(hunting): get_auth_request con 0x7fd0f00b4350 auth_method 0 2023-10-30T12:10:23.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5945640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5945640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5945640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5144640 10 monclient(hunting): get_auth_request con 0x7fd0f013eb50 auth_method 0 2023-10-30T12:10:23.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5144640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5144640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5144640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5945640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5945640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5945640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5144640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5144640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.395+0000 7fd0f5144640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.397+0000 7fd0f5945640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2023-10-30T12:10:23.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.397+0000 7fd0f5945640 1 -- 172.21.15.3:0/3994605105 >> v2:172.21.15.92:3300/0 conn(0x7fd0f0144f50 msgr2=0x7fd0f01453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:23.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.397+0000 7fd0f5945640 1 --2- 172.21.15.3:0/3994605105 >> v2:172.21.15.92:3300/0 conn(0x7fd0f0144f50 0x7fd0f01453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.397+0000 7fd0f5945640 1 -- 172.21.15.3:0/3994605105 >> v2:172.21.15.3:3300/0 conn(0x7fd0f013eb50 msgr2=0x7fd0f013ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:23.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.397+0000 7fd0f5945640 1 --2- 172.21.15.3:0/3994605105 >> v2:172.21.15.3:3300/0 conn(0x7fd0f013eb50 0x7fd0f013ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.397+0000 7fd0f5945640 10 monclient: _finish_hunting 0 2023-10-30T12:10:23.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.397+0000 7fd0f5945640 1 monclient: found mon.noname-b 2023-10-30T12:10:23.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.397+0000 7fd0f5945640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:23.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.397+0000 7fd0f5945640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.397+0000 7fd0f5945640 1 -- 172.21.15.3:0/3994605105 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f00a8910 con 0x7fd0f00b4350 2023-10-30T12:10:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.399+0000 7fd0f5945640 1 --2- 172.21.15.3:0/3994605105 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0f00b4350 0x7fd0f00a83d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd0f00a9770 tx=0x7fd0ec02b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5eacdf17d4232ca6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f4943640 1 -- 172.21.15.3:0/3994605105 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec031750 con 0x7fd0f00b4350 2023-10-30T12:10:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f4943640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f4943640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f4943640 10 monclient: dump: 2023-10-30T12:10:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.414 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f4943640 10 monclient: _finish_auth 0 2023-10-30T12:10:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f4943640 10 monclient: _check_auth_tickets 2023-10-30T12:10:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f4943640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f4943640 1 -- 172.21.15.3:0/3994605105 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ec0318c0 con 0x7fd0f00b4350 2023-10-30T12:10:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f7ba9640 5 monclient: authenticate success, global_id 4292 2023-10-30T12:10:23.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f7ba9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:23.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f4943640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:23.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f7ba9640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:23.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f4943640 1 -- 172.21.15.3:0/3994605105 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec031c60 con 0x7fd0f00b4350 2023-10-30T12:10:23.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f4943640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f7ba9640 4 set_mon_vals no callback set 2023-10-30T12:10:23.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f4943640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:23.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.403+0000 7fd0f4943640 10 monclient: dump: 2023-10-30T12:10:23.416 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.416 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.417 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.417 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.417 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.417 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.417 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.418 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.418 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.418 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.404+0000 7fd0f7ba9640 10 monclient: shutdown 2023-10-30T12:10:23.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.404+0000 7fd0f7ba9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:23.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.404+0000 7fd0f7ba9640 1 -- 172.21.15.3:0/3994605105 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0f00b4350 msgr2=0x7fd0f00a83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.404+0000 7fd0f7ba9640 1 --2- 172.21.15.3:0/3994605105 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0f00b4350 0x7fd0f00a83d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd0f00a9770 tx=0x7fd0ec02b7f0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.404+0000 7fd0f7ba9640 1 -- 172.21.15.3:0/3994605105 shutdown_connections 2023-10-30T12:10:23.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.404+0000 7fd0f7ba9640 1 --2- 172.21.15.3:0/3994605105 >> v2:172.21.15.92:3300/0 conn(0x7fd0f0144f50 0x7fd0f01453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.404+0000 7fd0f7ba9640 1 --2- 172.21.15.3:0/3994605105 >> v2:172.21.15.3:3300/0 conn(0x7fd0f013eb50 0x7fd0f013ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.404+0000 7fd0f7ba9640 1 --2- 172.21.15.3:0/3994605105 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0f00b4350 0x7fd0f00a83d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.404+0000 7fd0f7ba9640 1 -- 172.21.15.3:0/3994605105 >> 172.21.15.3:0/3994605105 conn(0x7fd0f0065ea0 msgr2=0x7fd0f00662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:23.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.404+0000 7fd0f7ba9640 1 -- 172.21.15.3:0/3994605105 shutdown_connections 2023-10-30T12:10:23.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.404+0000 7fd0f7ba9640 1 -- 172.21.15.3:0/3994605105 wait complete. 2023-10-30T12:10:23.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.405+0000 7fd0f7ba9640 10 monclient: build_initial_monmap 2023-10-30T12:10:23.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.405+0000 7fd0f7ba9640 10 monclient: monmap: 2023-10-30T12:10:23.421 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:23.421 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:23.421 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:23.421 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:23.421 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:23.422 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:23.422 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:23.422 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:23.422 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:23.422 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.405+0000 7fd0f7ba9640 1 librados: starting msgr at 2023-10-30T12:10:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.405+0000 7fd0f7ba9640 1 librados: starting objecter 2023-10-30T12:10:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.406+0000 7fd0f7ba9640 1 Processor -- start 2023-10-30T12:10:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.406+0000 7fd0f7ba9640 1 -- start start 2023-10-30T12:10:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.406+0000 7fd0f7ba9640 1 librados: setting wanted keys 2023-10-30T12:10:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.406+0000 7fd0f7ba9640 1 librados: calling monclient init 2023-10-30T12:10:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.406+0000 7fd0f7ba9640 10 monclient: init 2023-10-30T12:10:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.406+0000 7fd0f7ba9640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f7ba9640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f7ba9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0f00b4350 0x7fd0f00b7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f7ba9640 10 monclient(hunting): picked mon.noname-b con 0x7fd0f00b4350 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f7ba9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0f013eb50 0x7fd0f013f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f7ba9640 10 monclient(hunting): picked mon.noname-a con 0x7fd0f013eb50 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f7ba9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0f0144f50 0x7fd0f0141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f7ba9640 10 monclient(hunting): picked mon.noname-c con 0x7fd0f0144f50 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f7ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f7ba9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00a9300 con 0x7fd0f013eb50 2023-10-30T12:10:23.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f7ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f7ba9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00b6d80 con 0x7fd0f00b4350 2023-10-30T12:10:23.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f7ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f7ba9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00b66d0 con 0x7fd0f0144f50 2023-10-30T12:10:23.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f7ba9640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:23.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f7ba9640 10 monclient(hunting): authenticate will time out at 1414.621166s 2023-10-30T12:10:23.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f6146640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0f0144f50 0x7fd0f0141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.407+0000 7fd0f5144640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0f013eb50 0x7fd0f013f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f5144640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0f013eb50 0x7fd0f013f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38118/0 (socket says 172.21.15.3:38118) 2023-10-30T12:10:23.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f6146640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0f0144f50 0x7fd0f0141a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:48472/0 (socket says 172.21.15.3:48472) 2023-10-30T12:10:23.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f5144640 1 -- 172.21.15.3:0/2921881203 learned_addr learned my addr 172.21.15.3:0/2921881203 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:23.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f5144640 10 monclient(hunting): get_auth_request con 0x7fd0f013eb50 auth_method 0 2023-10-30T12:10:23.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f5144640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f5144640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f5144640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f6146640 10 monclient(hunting): get_auth_request con 0x7fd0f0144f50 auth_method 0 2023-10-30T12:10:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f6146640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f6146640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f6146640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f6146640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f6146640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f6146640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f5144640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f5144640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.408+0000 7fd0f5144640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.413+0000 7fd0f6146640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2023-10-30T12:10:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.413+0000 7fd0f6146640 1 -- 172.21.15.3:0/2921881203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0f00b4350 msgr2=0x7fd0f00b7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.413+0000 7fd0f6146640 1 --2- 172.21.15.3:0/2921881203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0f00b4350 0x7fd0f00b7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.413+0000 7fd0f6146640 1 -- 172.21.15.3:0/2921881203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0f013eb50 msgr2=0x7fd0f013f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.413+0000 7fd0f6146640 1 --2- 172.21.15.3:0/2921881203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0f013eb50 0x7fd0f013f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.413+0000 7fd0f6146640 10 monclient: _finish_hunting 0 2023-10-30T12:10:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.413+0000 7fd0f6146640 1 monclient: found mon.noname-c 2023-10-30T12:10:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.413+0000 7fd0f6146640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.413+0000 7fd0f6146640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.413+0000 7fd0f6146640 1 -- 172.21.15.3:0/2921881203 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f01d6a80 con 0x7fd0f0144f50 2023-10-30T12:10:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.413+0000 7fd0f5144640 1 --2- 172.21.15.3:0/2921881203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0f013eb50 0x7fd0f013f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:23.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.416+0000 7fd0f6146640 1 --2- 172.21.15.3:0/2921881203 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0f0144f50 0x7fd0f0141a10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800be20 tx=0x7fd0e800bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.416+0000 7fd0e67fc640 10 client.?.objecter ms_handle_connect 0x7fd0f0144f50 2023-10-30T12:10:23.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.416+0000 7fd0e67fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:23.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0e67fc640 1 -- 172.21.15.3:0/2921881203 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e800ce70 con 0x7fd0f0144f50 2023-10-30T12:10:23.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0e67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:23.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0e67fc640 10 monclient: dump: 2023-10-30T12:10:23.436 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.436 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.436 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.436 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.437 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.437 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.437 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.437 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.437 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.438 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0e67fc640 10 monclient: _finish_auth 0 2023-10-30T12:10:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0e67fc640 10 monclient: _check_auth_tickets 2023-10-30T12:10:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0e67fc640 1 -- 172.21.15.3:0/2921881203 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e8014440 con 0x7fd0f0144f50 2023-10-30T12:10:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0f7ba9640 5 monclient: authenticate success, global_id 4295 2023-10-30T12:10:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0f7ba9640 10 monclient: _renew_subs 2023-10-30T12:10:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0f7ba9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0f7ba9640 1 -- 172.21.15.3:0/2921881203 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f01402d0 con 0x7fd0f0144f50 2023-10-30T12:10:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0e67fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0e67fc640 1 -- 172.21.15.3:0/2921881203 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8014800 con 0x7fd0f0144f50 2023-10-30T12:10:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0e67fc640 10 monclient: dump: 2023-10-30T12:10:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.441 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.441 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.441 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.441 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.441 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.441 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.442 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.442 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0e7fff640 4 set_mon_vals no callback set 2023-10-30T12:10:23.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0f7ba9640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:23.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0f7ba9640 10 monclient: _renew_subs 2023-10-30T12:10:23.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0f7ba9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.419+0000 7fd0f7ba9640 1 -- 172.21.15.3:0/2921881203 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f01d73d0 con 0x7fd0f0144f50 2023-10-30T12:10:23.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.420+0000 7fd0f7ba9640 1 librados: init done 2023-10-30T12:10:23.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.420+0000 7fd0f7ba9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.420+0000 7fd0f7ba9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.420+0000 7fd0f7ba9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.420+0000 7fd0f7ba9640 1 -- 172.21.15.3:0/2921881203 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f004e9b0 con 0x7fd0f0144f50 2023-10-30T12:10:23.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.512+0000 7fc063fff640 1 -- 172.21.15.3:0/3261684186 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc06c05c1f0 con 0x7fc07c0b4350 2023-10-30T12:10:23.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.512+0000 7fc063fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.512+0000 7fc063fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:23.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.516+0000 7fc576ffd640 1 -- 172.21.15.3:0/2696475202 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc5680816e0 con 0x7fc5800b2920 2023-10-30T12:10:23.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.516+0000 7fc576ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T12:10:23.517 INFO:teuthology.orchestra.run.smithi003.stdout:60129542150 2023-10-30T12:10:23.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.516+0000 7fc576ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:23.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.518+0000 7fc5848cf640 10 librados: watch_flush enter 2023-10-30T12:10:23.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.518+0000 7fc5848cf640 10 librados: watch_flush exit 2023-10-30T12:10:23.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.518+0000 7fc5848cf640 20 client.4351.objecter shutdown clearing up homeless session... 2023-10-30T12:10:23.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.518+0000 7fc5848cf640 10 client.4351.objecter successfully canceled tick 2023-10-30T12:10:23.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.518+0000 7fc5848cf640 1 -- 172.21.15.3:0/2696475202 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc56006c7c0 msgr2=0x7fc56006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.518+0000 7fc5848cf640 1 --2- 172.21.15.3:0/2696475202 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc56006c7c0 0x7fc56006ec80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc578009bd0 tx=0x7fc578009240 comp rx=0 tx=0).stop 2023-10-30T12:10:23.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.518+0000 7fc5848cf640 10 monclient: shutdown 2023-10-30T12:10:23.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.518+0000 7fc5848cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:23.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.518+0000 7fc5848cf640 1 -- 172.21.15.3:0/2696475202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5800b2920 msgr2=0x7fc5801da560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.518+0000 7fc5848cf640 1 --2- 172.21.15.3:0/2696475202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5800b2920 0x7fc5801da560 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc568009890 tx=0x7fc5680046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.518+0000 7fc5848cf640 1 -- 172.21.15.3:0/2696475202 shutdown_connections 2023-10-30T12:10:23.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.518+0000 7fc5848cf640 1 --2- 172.21.15.3:0/2696475202 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc56006c7c0 0x7fc56006ec80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.518+0000 7fc5848cf640 1 --2- 172.21.15.3:0/2696475202 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc5800ad750 0x7fc580143e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.518+0000 7fc5848cf640 1 --2- 172.21.15.3:0/2696475202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5800b2920 0x7fc5801da560 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.518+0000 7fc5848cf640 1 --2- 172.21.15.3:0/2696475202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5800a8e40 0x7fc58013a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.519+0000 7fc5848cf640 1 -- 172.21.15.3:0/2696475202 >> 172.21.15.3:0/2696475202 conn(0x7fc5800b7180 msgr2=0x7fc5800b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:23.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.519+0000 7fc5848cf640 1 -- 172.21.15.3:0/2696475202 shutdown_connections 2023-10-30T12:10:23.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.519+0000 7fc5848cf640 1 -- 172.21.15.3:0/2696475202 wait complete. 2023-10-30T12:10:23.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.519+0000 7fc5848cf640 1 librados: shutdown 2023-10-30T12:10:23.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T12:10:23.535 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.7 2023-10-30T12:10:23.535 DEBUG:teuthology.parallel:result is None 2023-10-30T12:10:23.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T12:10:23.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.569+0000 7f38f9e1f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:10:23.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.569+0000 7f38f9e1f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:10:23.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.569+0000 7f38f9e1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.569+0000 7f38f9e1f640 1 -- 172.21.15.3:0/1721248034 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f38f4140620 con 0x7f38f41da760 2023-10-30T12:10:23.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T12:10:23.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.644+0000 7faed1597640 10 monclient: get_monmap_and_config 2023-10-30T12:10:23.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.644+0000 7faed1597640 10 monclient: build_initial_monmap 2023-10-30T12:10:23.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.644+0000 7faed1597640 10 monclient: monmap: 2023-10-30T12:10:23.647 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:23.647 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.647 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:23.645718+0000 2023-10-30T12:10:23.647 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:23.645718+0000 2023-10-30T12:10:23.647 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:23.648 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.648 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:23.648 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:23.648 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:23.648 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:23.649 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:23.649 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.645+0000 7faed1597640 1 Processor -- start 2023-10-30T12:10:23.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.645+0000 7faed1597640 1 -- start start 2023-10-30T12:10:23.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.645+0000 7faed1597640 10 monclient: init 2023-10-30T12:10:23.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.646+0000 7faed1597640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:23.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.646+0000 7faed1597640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-10-30T12:10:23.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.646+0000 7faed1597640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faecc0a8e40 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:23.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.646+0000 7faed1597640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7faecc0b13c0 0x7faecc141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.646+0000 7faed1597640 10 monclient(hunting): picked mon.noname-c con 0x7faecc0b13c0 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:23.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.646+0000 7faed1597640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faecc1423f0 0x7faecc14c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.646+0000 7faed1597640 10 monclient(hunting): picked mon.noname-a con 0x7faecc1423f0 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:23.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.646+0000 7faed1597640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc0b1b00 con 0x7faecc0a8e40 2023-10-30T12:10:23.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.646+0000 7faed1597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.646+0000 7faed1597640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faecc0679a0 con 0x7faecc1423f0 2023-10-30T12:10:23.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.646+0000 7faed1597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.646+0000 7faed1597640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faecc0b1060 con 0x7faecc0b13c0 2023-10-30T12:10:23.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.646+0000 7faed1597640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:23.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.646+0000 7faed1597640 10 monclient(hunting): authenticate will time out at 1414.860568s 2023-10-30T12:10:23.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.647+0000 7faeca7fc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7faecc0b13c0 0x7faecc141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.647+0000 7faeca7fc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7faecc0b13c0 0x7faecc141eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34044/0 (socket says 172.21.15.3:34044) 2023-10-30T12:10:23.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.647+0000 7faeca7fc640 1 -- 172.21.15.3:0/580578446 learned_addr learned my addr 172.21.15.3:0/580578446 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:23.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.647+0000 7faeca7fc640 10 monclient(hunting): get_auth_request con 0x7faecc0b13c0 auth_method 0 2023-10-30T12:10:23.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.647+0000 7faeca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.647+0000 7faeca7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.647+0000 7faeca7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.648+0000 7faeca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.648+0000 7faeca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.648+0000 7faeca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.648+0000 7faecb7fe640 1 --2- 172.21.15.3:0/580578446 >> v2:172.21.15.3:3300/0 conn(0x7faecc1423f0 0x7faecc14c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.649+0000 7faecb7fe640 10 monclient(hunting): get_auth_request con 0x7faecc1423f0 auth_method 0 2023-10-30T12:10:23.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.649+0000 7faecb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.649+0000 7faecb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.649+0000 7faecb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.649+0000 7faeca7fc640 10 monclient(hunting): handle_auth_done global_id 4363 payload 274 2023-10-30T12:10:23.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.649+0000 7faeca7fc640 1 -- 172.21.15.3:0/580578446 >> v2:172.21.15.3:3300/0 conn(0x7faecc1423f0 msgr2=0x7faecc14c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:23.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.649+0000 7faeca7fc640 1 --2- 172.21.15.3:0/580578446 >> v2:172.21.15.3:3300/0 conn(0x7faecc1423f0 0x7faecc14c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.649+0000 7faeca7fc640 1 -- 172.21.15.3:0/580578446 >> v1:172.21.15.3:6789/0 conn(0x7faecc0a8e40 legacy=0x7faecc13efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:23.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.649+0000 7faeca7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:10:23.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.649+0000 7faeca7fc640 1 monclient: found mon.noname-c 2023-10-30T12:10:23.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.649+0000 7faeca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:23.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.649+0000 7faeca7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:10:23.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.649+0000 7faeca7fc640 1 -- 172.21.15.3:0/580578446 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc0a8920 con 0x7faecc0b13c0 2023-10-30T12:10:23.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.652+0000 7faeca7fc640 1 --2- 172.21.15.3:0/580578446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc0b13c0 0x7faecc141eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faecc067880 tx=0x7faec002b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=718222a5e75ab2a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faec9ffb640 1 -- 172.21.15.3:0/580578446 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec0031750 con 0x7faecc0b13c0 2023-10-30T12:10:23.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faec9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faec9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:23.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faec9ffb640 10 monclient: dump: 2023-10-30T12:10:23.662 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.662 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.662 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.663 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.664 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faec9ffb640 10 monclient: _finish_auth 0 2023-10-30T12:10:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faec9ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:10:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faec9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:23.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faec9ffb640 1 -- 172.21.15.3:0/580578446 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec00318c0 con 0x7faecc0b13c0 2023-10-30T12:10:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faec9ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faec9ffb640 1 -- 172.21.15.3:0/580578446 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec0031c60 con 0x7faecc0b13c0 2023-10-30T12:10:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faec9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faec9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:23.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faec9ffb640 10 monclient: dump: 2023-10-30T12:10:23.666 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.666 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.666 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.666 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.666 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.667 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.667 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.667 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.667 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.667 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faed1597640 5 monclient: authenticate success, global_id 4363 2023-10-30T12:10:23.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faed1597640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:23.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.655+0000 7faed1597640 4 set_mon_vals no callback set 2023-10-30T12:10:23.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.656+0000 7faed1597640 10 monclient: shutdown 2023-10-30T12:10:23.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.656+0000 7faed1597640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:23.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.656+0000 7faed1597640 1 -- 172.21.15.3:0/580578446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc0b13c0 msgr2=0x7faecc141eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.656+0000 7faed1597640 1 --2- 172.21.15.3:0/580578446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc0b13c0 0x7faecc141eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faecc067880 tx=0x7faec002b7f0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.656+0000 7faed1597640 1 -- 172.21.15.3:0/580578446 shutdown_connections 2023-10-30T12:10:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.656+0000 7faed1597640 1 --2- 172.21.15.3:0/580578446 >> v2:172.21.15.3:3300/0 conn(0x7faecc1423f0 0x7faecc14c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.656+0000 7faed1597640 1 --2- 172.21.15.3:0/580578446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc0b13c0 0x7faecc141eb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.656+0000 7faed1597640 1 -- 172.21.15.3:0/580578446 >> 172.21.15.3:0/580578446 conn(0x7faecc0b7180 msgr2=0x7faecc0b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:23.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.656+0000 7faed1597640 1 -- 172.21.15.3:0/580578446 shutdown_connections 2023-10-30T12:10:23.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.656+0000 7faed1597640 1 -- 172.21.15.3:0/580578446 wait complete. 2023-10-30T12:10:23.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.657+0000 7faed1597640 10 monclient: build_initial_monmap 2023-10-30T12:10:23.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.657+0000 7faed1597640 10 monclient: monmap: 2023-10-30T12:10:23.670 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:23.671 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:23.671 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:23.671 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:23.671 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:23.671 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:23.672 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:23.672 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:23.672 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:23.672 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.658+0000 7faed1597640 1 librados: starting msgr at 2023-10-30T12:10:23.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.658+0000 7faed1597640 1 librados: starting objecter 2023-10-30T12:10:23.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.658+0000 7faed1597640 1 Processor -- start 2023-10-30T12:10:23.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.658+0000 7faed1597640 1 -- start start 2023-10-30T12:10:23.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.658+0000 7faed1597640 1 librados: setting wanted keys 2023-10-30T12:10:23.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.658+0000 7faed1597640 1 librados: calling monclient init 2023-10-30T12:10:23.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.658+0000 7faed1597640 10 monclient: init 2023-10-30T12:10:23.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:23.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:10:23.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faecc0a8e40 0x7faecc1459b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 10 monclient(hunting): picked mon.noname-c con 0x7faecc0a8e40 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:23.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc1423f0 0x7faecc0ab190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 10 monclient(hunting): picked mon.noname-b con 0x7faecc1423f0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:23.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faecc0acde0 0x7faecc1d7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 10 monclient(hunting): picked mon.noname-a con 0x7faecc0acde0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:23.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc0b4890 con 0x7faecc0acde0 2023-10-30T12:10:23.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc0aba60 con 0x7faecc1423f0 2023-10-30T12:10:23.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faecc0aca30 con 0x7faecc0a8e40 2023-10-30T12:10:23.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:23.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.659+0000 7faed1597640 10 monclient(hunting): authenticate will time out at 1414.873692s 2023-10-30T12:10:23.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.660+0000 7faeca7fc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc1423f0 0x7faecc0ab190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.660+0000 7faecb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faecc0acde0 0x7faecc1d7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.660+0000 7faeca7fc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc1423f0 0x7faecc0ab190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34058/0 (socket says 172.21.15.3:34058) 2023-10-30T12:10:23.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.660+0000 7faeca7fc640 1 -- 172.21.15.3:0/2514375206 learned_addr learned my addr 172.21.15.3:0/2514375206 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:23.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.660+0000 7faeca7fc640 10 monclient(hunting): get_auth_request con 0x7faecc1423f0 auth_method 0 2023-10-30T12:10:23.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.660+0000 7faeca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.660+0000 7faeca7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.660+0000 7faeca7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.660+0000 7faecb7fe640 10 monclient(hunting): get_auth_request con 0x7faecc0acde0 auth_method 0 2023-10-30T12:10:23.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.660+0000 7faecb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.660+0000 7faecb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.660+0000 7faecb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.660+0000 7faeca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.660+0000 7faeca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.660+0000 7faeca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.661+0000 7faecb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.661+0000 7faecb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.661+0000 7faecb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.663+0000 7faeca7fc640 10 monclient(hunting): handle_auth_done global_id 4366 payload 824 2023-10-30T12:10:23.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.663+0000 7faeca7fc640 1 -- 172.21.15.3:0/2514375206 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faecc0a8e40 msgr2=0x7faecc1459b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.663+0000 7faeca7fc640 1 --2- 172.21.15.3:0/2514375206 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faecc0a8e40 0x7faecc1459b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.663+0000 7faeca7fc640 1 -- 172.21.15.3:0/2514375206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faecc0acde0 msgr2=0x7faecc1d7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.663+0000 7faeca7fc640 1 --2- 172.21.15.3:0/2514375206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faecc0acde0 0x7faecc1d7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.663+0000 7faeca7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:10:23.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.663+0000 7faeca7fc640 1 monclient: found mon.noname-b 2023-10-30T12:10:23.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.663+0000 7faeca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:23.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.663+0000 7faeca7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:23.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.663+0000 7faeca7fc640 1 -- 172.21.15.3:0/2514375206 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc0ab6d0 con 0x7faecc1423f0 2023-10-30T12:10:23.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.665+0000 7faeca7fc640 1 --2- 172.21.15.3:0/2514375206 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc1423f0 0x7faecc0ab190 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faec0000c00 tx=0x7faec00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.665+0000 7faeb37fe640 10 client.?.objecter ms_handle_connect 0x7faecc1423f0 2023-10-30T12:10:23.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.665+0000 7faeb37fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:23.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.670+0000 7faeb37fe640 1 -- 172.21.15.3:0/2514375206 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec0033c80 con 0x7faecc1423f0 2023-10-30T12:10:23.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.670+0000 7faeb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.670+0000 7faeb37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:23.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.670+0000 7faeb37fe640 10 monclient: dump: 2023-10-30T12:10:23.685 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.685 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.685 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.685 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.686 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.686 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.686 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.686 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.686 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.687 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.670+0000 7faeb37fe640 10 monclient: _finish_auth 0 2023-10-30T12:10:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.670+0000 7faeb37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:10:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.670+0000 7faeb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.670+0000 7faeb37fe640 1 -- 172.21.15.3:0/2514375206 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec0033df0 con 0x7faecc1423f0 2023-10-30T12:10:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.670+0000 7faeb37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:23.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.670+0000 7faec97fa640 4 set_mon_vals no callback set 2023-10-30T12:10:23.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.671+0000 7faeb37fe640 1 -- 172.21.15.3:0/2514375206 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec003c620 con 0x7faecc1423f0 2023-10-30T12:10:23.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.671+0000 7faed1597640 5 monclient: authenticate success, global_id 4366 2023-10-30T12:10:23.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.671+0000 7faed1597640 10 monclient: _renew_subs 2023-10-30T12:10:23.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.671+0000 7faed1597640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:23.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.671+0000 7faed1597640 1 -- 172.21.15.3:0/2514375206 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc144b00 con 0x7faecc1423f0 2023-10-30T12:10:23.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.672+0000 7faed1597640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:23.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.672+0000 7faed1597640 10 monclient: _renew_subs 2023-10-30T12:10:23.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.672+0000 7faed1597640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:23.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.672+0000 7faed1597640 1 -- 172.21.15.3:0/2514375206 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faecc1d78b0 con 0x7faecc1423f0 2023-10-30T12:10:23.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.672+0000 7faed1597640 1 librados: init done 2023-10-30T12:10:23.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.672+0000 7faeb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.672+0000 7faeb37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:23.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.672+0000 7faeb37fe640 10 monclient: dump: 2023-10-30T12:10:23.691 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.691 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.691 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.691 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.691 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.692 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.692 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.692 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.692 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.692 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.673+0000 7faed1597640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.673+0000 7faed1597640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.673+0000 7faed1597640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:23.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.673+0000 7faed1597640 1 -- 172.21.15.3:0/2514375206 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faecc04e9b0 con 0x7faecc1423f0 2023-10-30T12:10:23.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.683+0000 7f88b28cf640 10 monclient: get_monmap_and_config 2023-10-30T12:10:23.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.683+0000 7f88b28cf640 10 monclient: build_initial_monmap 2023-10-30T12:10:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.683+0000 7f88b28cf640 10 monclient: monmap: 2023-10-30T12:10:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:23.684027+0000 2023-10-30T12:10:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:23.684027+0000 2023-10-30T12:10:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:23.698 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.683+0000 7f88b28cf640 1 Processor -- start 2023-10-30T12:10:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.683+0000 7f88b28cf640 1 -- start start 2023-10-30T12:10:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.683+0000 7f88b28cf640 10 monclient: init 2023-10-30T12:10:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.684+0000 7f88b28cf640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.684+0000 7f88b28cf640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-10-30T12:10:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.684+0000 7f88b28cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88ac0a8e40 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.684+0000 7f88b28cf640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f88ac0b13c0 0x7f88ac141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.684+0000 7f88b28cf640 10 monclient(hunting): picked mon.noname-b con 0x7f88ac0b13c0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:23.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.684+0000 7f88b28cf640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f88ac1423f0 0x7f88ac14c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.684+0000 7f88b28cf640 10 monclient(hunting): picked mon.noname-a con 0x7f88ac1423f0 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:23.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.684+0000 7f88b28cf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ac0b1b00 con 0x7f88ac0a8e40 2023-10-30T12:10:23.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.684+0000 7f88b28cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.684+0000 7f88b28cf640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ac0679a0 con 0x7f88ac1423f0 2023-10-30T12:10:23.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.684+0000 7f88b28cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.684+0000 7f88b28cf640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac0b1060 con 0x7f88ac0b13c0 2023-10-30T12:10:23.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.684+0000 7f88b28cf640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:23.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.684+0000 7f88b28cf640 10 monclient(hunting): authenticate will time out at 1414.898822s 2023-10-30T12:10:23.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.686+0000 7f88abfff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f88ac0a8e40 0x7f88ac13efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:39606/0 (socket says 172.21.15.3:39606) 2023-10-30T12:10:23.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.686+0000 7f88abfff640 1 -- 172.21.15.3:0/2200859139 learned_addr learned my addr 172.21.15.3:0/2200859139 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:23.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.686+0000 7f88b0e6c640 1 --2- 172.21.15.3:0/2200859139 >> v2:172.21.15.3:3300/0 conn(0x7f88ac1423f0 0x7f88ac14c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.687+0000 7f88b0e6c640 10 monclient(hunting): get_auth_request con 0x7f88ac1423f0 auth_method 0 2023-10-30T12:10:23.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.687+0000 7f88b0e6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.687+0000 7f88b0e6c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.687+0000 7f88b0e6c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.688+0000 7f88b0e6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.688+0000 7f88b0e6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.688+0000 7f88b0e6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.691+0000 7f88b0e6c640 10 monclient(hunting): handle_auth_done global_id 4392 payload 274 2023-10-30T12:10:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.691+0000 7f88b0e6c640 1 -- 172.21.15.3:0/2200859139 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f88ac0b13c0 msgr2=0x7f88ac141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:23.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.691+0000 7f88b0e6c640 1 --2- 172.21.15.3:0/2200859139 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f88ac0b13c0 0x7f88ac141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.691+0000 7f88b0e6c640 1 -- 172.21.15.3:0/2200859139 >> v1:172.21.15.3:6789/0 conn(0x7f88ac0a8e40 legacy=0x7f88ac13efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.691+0000 7f88b0e6c640 10 monclient: _finish_hunting 0 2023-10-30T12:10:23.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.691+0000 7f88b0e6c640 1 monclient: found mon.noname-a 2023-10-30T12:10:23.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.691+0000 7f88b0e6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:23.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.691+0000 7f88b0e6c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:23.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.691+0000 7f88b0e6c640 1 -- 172.21.15.3:0/2200859139 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ac0a8920 con 0x7f88ac1423f0 2023-10-30T12:10:23.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.692+0000 7faeb37fe640 1 -- 172.21.15.3:0/2514375206 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7faec003a040 con 0x7faecc1423f0 2023-10-30T12:10:23.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.693+0000 7faeb37fe640 1 --2- 172.21.15.3:0/2514375206 >> v2:172.21.15.92:6800/1750155735 conn(0x7faeb806c810 0x7faeb806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.693+0000 7faecaffd640 1 --2- 172.21.15.3:0/2514375206 >> v2:172.21.15.92:6800/1750155735 conn(0x7faeb806c810 0x7faeb806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.693+0000 7faecaffd640 10 monclient: get_auth_request con 0x7faeb806c810 auth_method 0 2023-10-30T12:10:23.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.694+0000 7faeb37fe640 10 client.4366.objecter ms_handle_connect 0x7faeb806c810 2023-10-30T12:10:23.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.694+0000 7faecaffd640 1 --2- 172.21.15.3:0/2514375206 >> v2:172.21.15.92:6800/1750155735 conn(0x7faeb806c810 0x7faeb806ecd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faebc001560 tx=0x7faebc007990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.695+0000 7faeb37fe640 1 -- 172.21.15.3:0/2514375206 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7faec0079ca0 con 0x7faecc1423f0 2023-10-30T12:10:23.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.695+0000 7faeb37fe640 10 client.4366.objecter ms_dispatch 0x7faecc000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:23.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.695+0000 7faeb37fe640 3 client.4366.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:23.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.695+0000 7faeb37fe640 3 client.4366.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:23.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.695+0000 7faeb37fe640 20 client.4366.objecter dump_active .. 0 homeless 2023-10-30T12:10:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.695+0000 7f88b0e6c640 1 --2- 172.21.15.3:0/2200859139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac1423f0 0x7f88ac14c320 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f88a4009d90 tx=0x7f88a402bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc53eaa804176 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88aaffd640 1 -- 172.21.15.3:0/2200859139 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a402cb40 con 0x7f88ac1423f0 2023-10-30T12:10:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88aaffd640 10 monclient: dump: 2023-10-30T12:10:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.714 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88aaffd640 10 monclient: _finish_auth 0 2023-10-30T12:10:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88aaffd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88aaffd640 1 -- 172.21.15.3:0/2200859139 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a402ccb0 con 0x7f88ac1423f0 2023-10-30T12:10:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88aaffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88aaffd640 1 -- 172.21.15.3:0/2200859139 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a4034660 con 0x7f88ac1423f0 2023-10-30T12:10:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88aaffd640 10 monclient: dump: 2023-10-30T12:10:23.716 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.716 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.716 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.716 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.716 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.716 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.717 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.717 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.717 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.717 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88b28cf640 5 monclient: authenticate success, global_id 4392 2023-10-30T12:10:23.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88b28cf640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:23.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.700+0000 7f88b28cf640 4 set_mon_vals no callback set 2023-10-30T12:10:23.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.701+0000 7f88b28cf640 10 monclient: shutdown 2023-10-30T12:10:23.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.701+0000 7f88b28cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:23.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.701+0000 7f88b28cf640 1 -- 172.21.15.3:0/2200859139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac1423f0 msgr2=0x7f88ac14c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.701+0000 7f88b28cf640 1 --2- 172.21.15.3:0/2200859139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac1423f0 0x7f88ac14c320 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f88a4009d90 tx=0x7f88a402bd90 comp rx=0 tx=0).stop 2023-10-30T12:10:23.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.701+0000 7f88b28cf640 1 -- 172.21.15.3:0/2200859139 shutdown_connections 2023-10-30T12:10:23.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.701+0000 7f88b28cf640 1 --2- 172.21.15.3:0/2200859139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac1423f0 0x7f88ac14c320 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.701+0000 7f88b28cf640 1 --2- 172.21.15.3:0/2200859139 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f88ac0b13c0 0x7f88ac141eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.701+0000 7f88b28cf640 1 -- 172.21.15.3:0/2200859139 >> 172.21.15.3:0/2200859139 conn(0x7f88ac0b7180 msgr2=0x7f88ac0b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:23.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.701+0000 7f88b28cf640 1 -- 172.21.15.3:0/2200859139 shutdown_connections 2023-10-30T12:10:23.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.701+0000 7f88b28cf640 1 -- 172.21.15.3:0/2200859139 wait complete. 2023-10-30T12:10:23.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.702+0000 7f88b28cf640 10 monclient: build_initial_monmap 2023-10-30T12:10:23.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.702+0000 7f88b28cf640 10 monclient: monmap: 2023-10-30T12:10:23.720 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:23.720 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:23.721 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:23.721 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:23.721 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:23.721 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:23.721 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:23.722 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:23.722 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:23.722 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.703+0000 7f88b28cf640 1 librados: starting msgr at 2023-10-30T12:10:23.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.703+0000 7f88b28cf640 1 librados: starting objecter 2023-10-30T12:10:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.703+0000 7f88b28cf640 1 Processor -- start 2023-10-30T12:10:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.703+0000 7f88b28cf640 1 -- start start 2023-10-30T12:10:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.703+0000 7f88b28cf640 1 librados: setting wanted keys 2023-10-30T12:10:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.703+0000 7f88b28cf640 1 librados: calling monclient init 2023-10-30T12:10:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.703+0000 7f88b28cf640 10 monclient: init 2023-10-30T12:10:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:23.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:10:23.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f88ac0a8e40 0x7f88ac1da400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 10 monclient(hunting): picked mon.noname-c con 0x7f88ac0a8e40 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:23.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac1423f0 0x7f88ac121460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 10 monclient(hunting): picked mon.noname-a con 0x7f88ac1423f0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:23.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88ac1dae20 0x7f88ac13d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 10 monclient(hunting): picked mon.noname-b con 0x7f88ac1dae20 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:23.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac0a83d0 con 0x7f88ac1423f0 2023-10-30T12:10:23.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac121b10 con 0x7f88ac1dae20 2023-10-30T12:10:23.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac11edc0 con 0x7f88ac0a8e40 2023-10-30T12:10:23.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:23.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.704+0000 7f88b28cf640 10 monclient(hunting): authenticate will time out at 1414.918332s 2023-10-30T12:10:23.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.705+0000 7f88ab7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac1423f0 0x7f88ac121460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.705+0000 7fd0e67fc640 1 -- 172.21.15.3:0/2921881203 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fd0e801d3b0 con 0x7fd0f0144f50 2023-10-30T12:10:23.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.705+0000 7f88ab7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac1423f0 0x7f88ac121460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38146/0 (socket says 172.21.15.3:38146) 2023-10-30T12:10:23.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.705+0000 7f88ab7fe640 1 -- 172.21.15.3:0/2783882881 learned_addr learned my addr 172.21.15.3:0/2783882881 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:23.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.705+0000 7f88ab7fe640 10 monclient(hunting): get_auth_request con 0x7f88ac1423f0 auth_method 0 2023-10-30T12:10:23.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.705+0000 7f88ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.705+0000 7f88ab7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.705+0000 7f88ab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.706+0000 7fd0e67fc640 1 --2- 172.21.15.3:0/2921881203 >> v2:172.21.15.92:6800/1750155735 conn(0x7fd0d406c810 0x7fd0d406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.706+0000 7fd0f5945640 1 --2- 172.21.15.3:0/2921881203 >> v2:172.21.15.92:6800/1750155735 conn(0x7fd0d406c810 0x7fd0d406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.706+0000 7fd0f5945640 10 monclient: get_auth_request con 0x7fd0d406c810 auth_method 0 2023-10-30T12:10:23.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.706+0000 7f88ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.706+0000 7f88ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.706+0000 7f88ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.706+0000 7fd0f5945640 1 --2- 172.21.15.3:0/2921881203 >> v2:172.21.15.92:6800/1750155735 conn(0x7fd0d406c810 0x7fd0d406ecd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec035000 tx=0x7fd0ec030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.706+0000 7fd0e67fc640 10 client.4295.objecter ms_handle_connect 0x7fd0d406c810 2023-10-30T12:10:23.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.709+0000 7f88ab7fe640 10 monclient(hunting): handle_auth_done global_id 4395 payload 824 2023-10-30T12:10:23.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.709+0000 7f88ab7fe640 1 -- 172.21.15.3:0/2783882881 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f88ac0a8e40 msgr2=0x7f88ac1da400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.709+0000 7f88ab7fe640 1 --2- 172.21.15.3:0/2783882881 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f88ac0a8e40 0x7f88ac1da400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.709+0000 7f88ab7fe640 1 -- 172.21.15.3:0/2783882881 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88ac1dae20 msgr2=0x7f88ac13d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.709+0000 7f88ab7fe640 1 --2- 172.21.15.3:0/2783882881 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88ac1dae20 0x7f88ac13d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.709+0000 7f88ab7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:10:23.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.709+0000 7f88ab7fe640 1 monclient: found mon.noname-a 2023-10-30T12:10:23.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.709+0000 7f88ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:23.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.709+0000 7f88ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:23.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.709+0000 7f88ab7fe640 1 -- 172.21.15.3:0/2783882881 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ac1208b0 con 0x7f88ac1423f0 2023-10-30T12:10:23.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.712+0000 7f88ab7fe640 1 --2- 172.21.15.3:0/2783882881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac1423f0 0x7f88ac121460 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f88a000b4a0 tx=0x7f88a000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.712+0000 7f88a8ff9640 10 client.?.objecter ms_handle_connect 0x7f88ac1423f0 2023-10-30T12:10:23.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.712+0000 7f88a8ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:23.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88a8ff9640 1 -- 172.21.15.3:0/2783882881 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0010610 con 0x7f88ac1423f0 2023-10-30T12:10:23.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88a8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:23.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88a8ff9640 10 monclient: dump: 2023-10-30T12:10:23.736 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.736 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.736 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.736 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.736 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.737 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.737 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.737 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.737 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.737 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88a8ff9640 10 monclient: _finish_auth 0 2023-10-30T12:10:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88a8ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:10:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88a8ff9640 1 -- 172.21.15.3:0/2783882881 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a0010780 con 0x7f88ac1423f0 2023-10-30T12:10:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88b28cf640 5 monclient: authenticate success, global_id 4395 2023-10-30T12:10:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88b28cf640 10 monclient: _renew_subs 2023-10-30T12:10:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88b28cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88b28cf640 1 -- 172.21.15.3:0/2783882881 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ac13d950 con 0x7f88ac1423f0 2023-10-30T12:10:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88a8ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88a8ff9640 1 -- 172.21.15.3:0/2783882881 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0010b40 con 0x7f88ac1423f0 2023-10-30T12:10:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88a8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:23.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88a8ff9640 10 monclient: dump: 2023-10-30T12:10:23.741 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.743 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.743 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.743 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.743 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88aa7fc640 4 set_mon_vals no callback set 2023-10-30T12:10:23.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88b28cf640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:23.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88b28cf640 10 monclient: _renew_subs 2023-10-30T12:10:23.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88b28cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:23.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.716+0000 7f88b28cf640 1 -- 172.21.15.3:0/2783882881 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ac13de90 con 0x7f88ac1423f0 2023-10-30T12:10:23.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.717+0000 7f88b28cf640 1 librados: init done 2023-10-30T12:10:23.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.718+0000 7f88b28cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.718+0000 7f88b28cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.718+0000 7f88b28cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:23.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.718+0000 7f88b28cf640 1 -- 172.21.15.3:0/2783882881 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ac0a9220 con 0x7f88ac1423f0 2023-10-30T12:10:23.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.721+0000 7fd0e67fc640 1 -- 172.21.15.3:0/2921881203 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fd0e8058660 con 0x7fd0f0144f50 2023-10-30T12:10:23.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.721+0000 7fd0e67fc640 10 client.4295.objecter ms_dispatch 0x7fd0f0000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:23.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.721+0000 7fd0e67fc640 3 client.4295.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:23.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.721+0000 7fd0e67fc640 3 client.4295.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:23.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.721+0000 7fd0e67fc640 20 client.4295.objecter dump_active .. 0 homeless 2023-10-30T12:10:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.721+0000 7fd0e67fc640 1 -- 172.21.15.3:0/2921881203 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd0f01d6a80 con 0x7fd0f0144f50 2023-10-30T12:10:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.721+0000 7fd0e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.721+0000 7fd0e67fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.738+0000 7fc082234640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T12:10:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.738+0000 7fc082234640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T12:10:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.738+0000 7fc082234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.738+0000 7fc082234640 1 -- 172.21.15.3:0/3261684186 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc07c1419d0 con 0x7fc07c0b4350 2023-10-30T12:10:23.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.745+0000 7f409cc7c640 10 monclient: get_monmap_and_config 2023-10-30T12:10:23.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.745+0000 7f409cc7c640 10 monclient: build_initial_monmap 2023-10-30T12:10:23.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.745+0000 7f409cc7c640 10 monclient: monmap: 2023-10-30T12:10:23.750 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:23.750 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.750 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:23.746817+0000 2023-10-30T12:10:23.750 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:23.746817+0000 2023-10-30T12:10:23.750 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:23.751 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.751 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:23.751 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:23.751 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:23.751 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:23.751 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:23.752 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.746+0000 7f409cc7c640 1 Processor -- start 2023-10-30T12:10:23.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.746+0000 7f409cc7c640 1 -- start start 2023-10-30T12:10:23.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.746+0000 7f409cc7c640 10 monclient: init 2023-10-30T12:10:23.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f409cc7c640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:23.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f409cc7c640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2023-10-30T12:10:23.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f409cc7c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f409806bc40 0x7f409806c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f409cc7c640 10 monclient(hunting): picked mon.noname-a con 0x7f409806bc40 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:23.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f38dbfff640 1 -- 172.21.15.3:0/1721248034 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f38ec006e50 con 0x7f38f41da760 2023-10-30T12:10:23.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f38dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:10:23.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f38dbfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:23.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f409cc7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4098071360 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:23.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f409cc7c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4098145190 0x7f4098145600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f409cc7c640 10 monclient(hunting): picked mon.noname-b con 0x7f4098145190 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:23.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f409cc7c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40980af8f0 con 0x7f4098071360 2023-10-30T12:10:23.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f409cc7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f409cc7c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40980679f0 con 0x7f409806bc40 2023-10-30T12:10:23.756 INFO:teuthology.orchestra.run.smithi003.stdout:68719476743 2023-10-30T12:10:23.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f409cc7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f409cc7c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f409806c560 con 0x7f4098145190 2023-10-30T12:10:23.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f409cc7c640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:23.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.747+0000 7f409cc7c640 10 monclient(hunting): authenticate will time out at 1414.961683s 2023-10-30T12:10:23.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.748+0000 7f88a8ff9640 1 -- 172.21.15.3:0/2783882881 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f88a0010cb0 con 0x7f88ac1423f0 2023-10-30T12:10:23.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.748+0000 7f88a8ff9640 1 --2- 172.21.15.3:0/2783882881 >> v2:172.21.15.92:6800/1750155735 conn(0x7f889806c7c0 0x7f889806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.749+0000 7f88abfff640 1 --2- 172.21.15.3:0/2783882881 >> v2:172.21.15.92:6800/1750155735 conn(0x7f889806c7c0 0x7f889806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.749+0000 7f88abfff640 10 monclient: get_auth_request con 0x7f889806c7c0 auth_method 0 2023-10-30T12:10:23.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.749+0000 7f88a8ff9640 10 client.4395.objecter ms_handle_connect 0x7f889806c7c0 2023-10-30T12:10:23.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.749+0000 7f88abfff640 1 --2- 172.21.15.3:0/2783882881 >> v2:172.21.15.92:6800/1750155735 conn(0x7f889806c7c0 0x7f889806ec80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f889c0015f0 tx=0x7f889c007a40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.749+0000 7f38f9e1f640 10 librados: watch_flush enter 2023-10-30T12:10:23.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.749+0000 7f38f9e1f640 10 librados: watch_flush exit 2023-10-30T12:10:23.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.749+0000 7f38f9e1f640 20 client.4289.objecter shutdown clearing up homeless session... 2023-10-30T12:10:23.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.749+0000 7f38f9e1f640 10 client.4289.objecter successfully canceled tick 2023-10-30T12:10:23.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.749+0000 7f38f9e1f640 1 -- 172.21.15.3:0/1721248034 >> v2:172.21.15.92:6800/1750155735 conn(0x7f38e006c7c0 msgr2=0x7f38e006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.749+0000 7f38f9e1f640 1 --2- 172.21.15.3:0/1721248034 >> v2:172.21.15.92:6800/1750155735 conn(0x7f38e006c7c0 0x7f38e006ec80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f38e4001780 tx=0x7f38e4009040 comp rx=0 tx=0).stop 2023-10-30T12:10:23.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.749+0000 7f38f9e1f640 10 monclient: shutdown 2023-10-30T12:10:23.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.749+0000 7f38f9e1f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:23.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.749+0000 7f38f9e1f640 1 -- 172.21.15.3:0/1721248034 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38f41da760 msgr2=0x7f38f41d66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.749+0000 7f38f9e1f640 1 --2- 172.21.15.3:0/1721248034 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38f41da760 0x7f38f41d66b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f38ec035680 tx=0x7f38ec004b50 comp rx=0 tx=0).stop 2023-10-30T12:10:23.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.750+0000 7f38f9e1f640 1 -- 172.21.15.3:0/1721248034 shutdown_connections 2023-10-30T12:10:23.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.750+0000 7f38f9e1f640 1 --2- 172.21.15.3:0/1721248034 >> v2:172.21.15.92:6800/1750155735 conn(0x7f38e006c7c0 0x7f38e006ec80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.750+0000 7f38f9e1f640 1 --2- 172.21.15.3:0/1721248034 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38f41da760 0x7f38f41d66b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.750+0000 7f38f9e1f640 1 --2- 172.21.15.3:0/1721248034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38f40ad640 0x7f38f41da220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.750+0000 7f38f9e1f640 1 --2- 172.21.15.3:0/1721248034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38dc005330 0x7f38f40ad100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.750+0000 7f38f9e1f640 1 -- 172.21.15.3:0/1721248034 >> 172.21.15.3:0/1721248034 conn(0x7f38f4066470 msgr2=0x7f38f4144430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:23.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.750+0000 7f38f9e1f640 1 -- 172.21.15.3:0/1721248034 shutdown_connections 2023-10-30T12:10:23.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.750+0000 7f38f9e1f640 1 -- 172.21.15.3:0/1721248034 wait complete. 2023-10-30T12:10:23.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.750+0000 7f38f9e1f640 1 librados: shutdown 2023-10-30T12:10:23.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.755+0000 7f4096d9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4098145190 0x7f4098145600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.757+0000 7f4096d9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4098145190 0x7f4098145600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:48514/0 (socket says 172.21.15.3:48514) 2023-10-30T12:10:23.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.757+0000 7f4096d9d640 1 -- 172.21.15.3:0/170363258 learned_addr learned my addr 172.21.15.3:0/170363258 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:23.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.757+0000 7f4096d9d640 10 monclient(hunting): get_auth_request con 0x7f4098145190 auth_method 0 2023-10-30T12:10:23.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.757+0000 7f4096d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.757+0000 7f4096d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.757+0000 7f4096d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.758+0000 7f4096d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.758+0000 7f4096d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.758+0000 7f4096d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.767 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476743 for osd.2 2023-10-30T12:10:23.767 DEBUG:teuthology.parallel:result is None 2023-10-30T12:10:23.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.761+0000 7f4096d9d640 10 monclient(hunting): handle_auth_done global_id 4298 payload 274 2023-10-30T12:10:23.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.761+0000 7f4096d9d640 1 -- 172.21.15.3:0/170363258 >> v2:172.21.15.3:3300/0 conn(0x7f409806bc40 msgr2=0x7f409806c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:23.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.761+0000 7f4096d9d640 1 --2- 172.21.15.3:0/170363258 >> v2:172.21.15.3:3300/0 conn(0x7f409806bc40 0x7f409806c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.762+0000 7f4096d9d640 1 -- 172.21.15.3:0/170363258 >> v1:172.21.15.3:6789/0 conn(0x7f4098071360 legacy=0x7f40980717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:23.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.762+0000 7f4096d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:10:23.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.762+0000 7f4096d9d640 1 monclient: found mon.noname-b 2023-10-30T12:10:23.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.762+0000 7f4096d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:23.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.762+0000 7f4096d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.762+0000 7f4096d9d640 1 -- 172.21.15.3:0/170363258 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409806f250 con 0x7f4098145190 2023-10-30T12:10:23.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.764+0000 7f4096d9d640 1 --2- 172.21.15.3:0/170363258 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4098145190 0x7f4098145600 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4090009e50 tx=0x7f409002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69bdc5d5ad1924ec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f4095d9b640 1 -- 172.21.15.3:0/170363258 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409002ca70 con 0x7f4098145190 2023-10-30T12:10:23.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f4095d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f4095d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:23.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f4095d9b640 10 monclient: dump: 2023-10-30T12:10:23.857 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.858 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.858 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.858 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.858 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.858 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.859 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.859 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.859 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.859 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f4095d9b640 10 monclient: _finish_auth 0 2023-10-30T12:10:23.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f4095d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:10:23.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f4095d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:23.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f4095d9b640 1 -- 172.21.15.3:0/170363258 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409002cbe0 con 0x7f4098145190 2023-10-30T12:10:23.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f4095d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:23.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f4095d9b640 1 -- 172.21.15.3:0/170363258 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40900343f0 con 0x7f4098145190 2023-10-30T12:10:23.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f409cc7c640 5 monclient: authenticate success, global_id 4298 2023-10-30T12:10:23.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f409cc7c640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:23.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f409cc7c640 4 set_mon_vals no callback set 2023-10-30T12:10:23.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f4095d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f4095d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:23.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.855+0000 7f4095d9b640 10 monclient: dump: 2023-10-30T12:10:23.862 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.862 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.862 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.862 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.862 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.863 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.863 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.863 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.863 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.863 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.856+0000 7f409cc7c640 10 monclient: shutdown 2023-10-30T12:10:23.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.856+0000 7f409cc7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:23.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.856+0000 7f409cc7c640 1 -- 172.21.15.3:0/170363258 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4098145190 msgr2=0x7f4098145600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.856+0000 7f409cc7c640 1 --2- 172.21.15.3:0/170363258 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4098145190 0x7f4098145600 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4090009e50 tx=0x7f409002bed0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.856+0000 7f409cc7c640 1 -- 172.21.15.3:0/170363258 shutdown_connections 2023-10-30T12:10:23.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.856+0000 7f409cc7c640 1 --2- 172.21.15.3:0/170363258 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4098145190 0x7f4098145600 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.856+0000 7f409cc7c640 1 --2- 172.21.15.3:0/170363258 >> v2:172.21.15.3:3300/0 conn(0x7f409806bc40 0x7f409806c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.856+0000 7f409cc7c640 1 -- 172.21.15.3:0/170363258 >> 172.21.15.3:0/170363258 conn(0x7f4098066470 msgr2=0x7f40980b4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:23.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.856+0000 7f409cc7c640 1 -- 172.21.15.3:0/170363258 shutdown_connections 2023-10-30T12:10:23.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.856+0000 7f409cc7c640 1 -- 172.21.15.3:0/170363258 wait complete. 2023-10-30T12:10:23.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.857+0000 7f409cc7c640 10 monclient: build_initial_monmap 2023-10-30T12:10:23.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.857+0000 7f409cc7c640 10 monclient: monmap: 2023-10-30T12:10:23.868 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:23.869 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:23.869 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:23.869 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:23.869 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:23.869 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:23.870 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:23.870 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:23.870 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:23.870 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.857+0000 7f409cc7c640 1 librados: starting msgr at 2023-10-30T12:10:23.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.857+0000 7f409cc7c640 1 librados: starting objecter 2023-10-30T12:10:23.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.857+0000 7f409cc7c640 1 Processor -- start 2023-10-30T12:10:23.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.857+0000 7f409cc7c640 1 -- start start 2023-10-30T12:10:23.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.857+0000 7f409cc7c640 1 librados: setting wanted keys 2023-10-30T12:10:23.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.857+0000 7f409cc7c640 1 librados: calling monclient init 2023-10-30T12:10:23.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.857+0000 7f409cc7c640 10 monclient: init 2023-10-30T12:10:23.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:23.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:23.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f409806bc40 0x7f409813f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 10 monclient(hunting): picked mon.noname-b con 0x7f409806bc40 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:23.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4098071360 0x7f40981da240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 10 monclient(hunting): picked mon.noname-a con 0x7f4098071360 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:23.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4098145190 0x7f40981d66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 10 monclient(hunting): picked mon.noname-c con 0x7f4098145190 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:23.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409813f450 con 0x7f4098071360 2023-10-30T12:10:23.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40981406c0 con 0x7f409806bc40 2023-10-30T12:10:23.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:23.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40981d9d50 con 0x7f4098145190 2023-10-30T12:10:23.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:23.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.858+0000 7f409cc7c640 10 monclient(hunting): authenticate will time out at 1415.072617s 2023-10-30T12:10:23.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.859+0000 7f4096d9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4098145190 0x7f40981d66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.860+0000 7f4096d9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4098145190 0x7f40981d66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:48526/0 (socket says 172.21.15.3:48526) 2023-10-30T12:10:23.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.860+0000 7f4096d9d640 1 -- 172.21.15.3:0/1349513910 learned_addr learned my addr 172.21.15.3:0/1349513910 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:23.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.860+0000 7f4096d9d640 10 monclient(hunting): get_auth_request con 0x7f4098145190 auth_method 0 2023-10-30T12:10:23.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.860+0000 7f4096d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:23.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.860+0000 7f4096d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:23.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.860+0000 7f4096d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:23.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.860+0000 7f4096d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:23.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.860+0000 7f4096d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:23.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.860+0000 7f4096d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:23.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.863+0000 7f4096d9d640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2023-10-30T12:10:23.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.863+0000 7f4096d9d640 1 -- 172.21.15.3:0/1349513910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f409806bc40 msgr2=0x7f409813f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.863+0000 7f4096d9d640 1 --2- 172.21.15.3:0/1349513910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f409806bc40 0x7f409813f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.863+0000 7f4096d9d640 1 -- 172.21.15.3:0/1349513910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4098071360 msgr2=0x7f40981da240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.863+0000 7f4096d9d640 1 --2- 172.21.15.3:0/1349513910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4098071360 0x7f40981da240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.863+0000 7f4096d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:10:23.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.863+0000 7f4096d9d640 1 monclient: found mon.noname-c 2023-10-30T12:10:23.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.864+0000 7f4096d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:23.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.864+0000 7f4096d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.864+0000 7f4096d9d640 1 -- 172.21.15.3:0/1349513910 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40980ad9a0 con 0x7f4098145190 2023-10-30T12:10:23.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.866+0000 7f4096d9d640 1 --2- 172.21.15.3:0/1349513910 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4098145190 0x7f40981d66a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4090009e50 tx=0x7f4090004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.866+0000 7f408effd640 10 client.?.objecter ms_handle_connect 0x7f4098145190 2023-10-30T12:10:23.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.866+0000 7f408effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:23.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.871+0000 7f408effd640 1 -- 172.21.15.3:0/1349513910 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4090040670 con 0x7f4098145190 2023-10-30T12:10:23.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.871+0000 7f408effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.871+0000 7f408effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:23.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.871+0000 7f408effd640 10 monclient: dump: 2023-10-30T12:10:23.882 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.882 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.883 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.883 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.883 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.883 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.883 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.884 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.884 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.884 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.871+0000 7f408effd640 10 monclient: _finish_auth 0 2023-10-30T12:10:23.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.871+0000 7f408effd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:23.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.871+0000 7f408effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:23.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.871+0000 7f408effd640 1 -- 172.21.15.3:0/1349513910 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40900407e0 con 0x7f4098145190 2023-10-30T12:10:23.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f408effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:23.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f409cc7c640 5 monclient: authenticate success, global_id 4301 2023-10-30T12:10:23.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f408effd640 1 -- 172.21.15.3:0/1349513910 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4090040ba0 con 0x7f4098145190 2023-10-30T12:10:23.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f408effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:23.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f408effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:23.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f408effd640 10 monclient: dump: 2023-10-30T12:10:23.886 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:23.886 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:23.887 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.887 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:23.887 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:23.887 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:23.887 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:23.888 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:23.888 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:23.888 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:23.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f409cc7c640 10 monclient: _renew_subs 2023-10-30T12:10:23.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f409cc7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f409cc7c640 1 -- 172.21.15.3:0/1349513910 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40980aec50 con 0x7f4098145190 2023-10-30T12:10:23.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f409559a640 4 set_mon_vals no callback set 2023-10-30T12:10:23.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f409cc7c640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:23.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f409cc7c640 10 monclient: _renew_subs 2023-10-30T12:10:23.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f409cc7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f409cc7c640 1 -- 172.21.15.3:0/1349513910 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4098140100 con 0x7f4098145190 2023-10-30T12:10:23.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f409cc7c640 1 librados: init done 2023-10-30T12:10:23.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f409cc7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f409cc7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f409cc7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.872+0000 7f409cc7c640 1 -- 172.21.15.3:0/1349513910 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409806cab0 con 0x7f4098145190 2023-10-30T12:10:23.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.891+0000 7f408effd640 1 -- 172.21.15.3:0/1349513910 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f409002ca80 con 0x7f4098145190 2023-10-30T12:10:23.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.891+0000 7f408effd640 1 --2- 172.21.15.3:0/1349513910 >> v2:172.21.15.92:6800/1750155735 conn(0x7f407806c7f0 0x7f407806ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:23.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.892+0000 7f409659c640 1 --2- 172.21.15.3:0/1349513910 >> v2:172.21.15.92:6800/1750155735 conn(0x7f407806c7f0 0x7f407806ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:23.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.892+0000 7f409659c640 10 monclient: get_auth_request con 0x7f407806c7f0 auth_method 0 2023-10-30T12:10:23.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.892+0000 7f409659c640 1 --2- 172.21.15.3:0/1349513910 >> v2:172.21.15.92:6800/1750155735 conn(0x7f407806c7f0 0x7f407806ecb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4088009710 tx=0x7f4088006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:23.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.892+0000 7f408effd640 10 client.4301.objecter ms_handle_connect 0x7f407806c7f0 2023-10-30T12:10:23.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.897+0000 7f408effd640 1 -- 172.21.15.3:0/1349513910 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f40900333a0 con 0x7f4098145190 2023-10-30T12:10:23.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.897+0000 7f408effd640 10 client.4301.objecter ms_dispatch 0x7f4098000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:23.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.897+0000 7f408effd640 3 client.4301.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:23.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.897+0000 7f408effd640 3 client.4301.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:23.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.897+0000 7f408effd640 20 client.4301.objecter dump_active .. 0 homeless 2023-10-30T12:10:23.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.925+0000 7faeb37fe640 1 -- 172.21.15.3:0/2514375206 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faec002f050 con 0x7faecc1423f0 2023-10-30T12:10:23.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.926+0000 7faeb37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:23.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.926+0000 7faeb37fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:23.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.942+0000 7fd0f7ba9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T12:10:23.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.942+0000 7fd0f7ba9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T12:10:23.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.942+0000 7fd0f7ba9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:23.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.942+0000 7fd0f7ba9640 1 -- 172.21.15.3:0/2921881203 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd0f01402d0 con 0x7fd0f0144f50 2023-10-30T12:10:23.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.990+0000 7fc063fff640 1 -- 172.21.15.3:0/3261684186 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc06c0603a0 con 0x7fc07c0b4350 2023-10-30T12:10:23.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.990+0000 7fc063fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T12:10:23.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.990+0000 7fc063fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:23.992 INFO:teuthology.orchestra.run.smithi003.stdout:68719476741 2023-10-30T12:10:23.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.992+0000 7fc082234640 10 librados: watch_flush enter 2023-10-30T12:10:23.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.992+0000 7fc082234640 10 librados: watch_flush exit 2023-10-30T12:10:23.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.992+0000 7fc082234640 20 client.4360.objecter shutdown clearing up homeless session... 2023-10-30T12:10:23.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.992+0000 7fc082234640 10 client.4360.objecter successfully canceled tick 2023-10-30T12:10:23.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.992+0000 7fc082234640 1 -- 172.21.15.3:0/3261684186 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc06806c810 msgr2=0x7fc06806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.992+0000 7fc082234640 1 --2- 172.21.15.3:0/3261684186 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc06806c810 0x7fc06806ecd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc070035000 tx=0x7fc070008030 comp rx=0 tx=0).stop 2023-10-30T12:10:23.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.992+0000 7fc082234640 10 monclient: shutdown 2023-10-30T12:10:23.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.992+0000 7fc082234640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:23.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.992+0000 7fc082234640 1 -- 172.21.15.3:0/3261684186 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07c0b4350 msgr2=0x7fc07c0b0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.992+0000 7fc082234640 1 --2- 172.21.15.3:0/3261684186 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07c0b4350 0x7fc07c0b0a30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00b4d0 tx=0x7fc06c00b990 comp rx=0 tx=0).stop 2023-10-30T12:10:23.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.993+0000 7fc082234640 1 -- 172.21.15.3:0/3261684186 shutdown_connections 2023-10-30T12:10:23.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.993+0000 7fc082234640 1 --2- 172.21.15.3:0/3261684186 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc06806c810 0x7fc06806ecd0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.993+0000 7fc082234640 1 --2- 172.21.15.3:0/3261684186 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc07c144f50 0x7fc07c1d6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.993+0000 7fc082234640 1 --2- 172.21.15.3:0/3261684186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc07c13eb50 0x7fc07c0b1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.993+0000 7fc082234640 1 --2- 172.21.15.3:0/3261684186 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07c0b4350 0x7fc07c0b0a30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.993+0000 7fc082234640 1 -- 172.21.15.3:0/3261684186 >> 172.21.15.3:0/3261684186 conn(0x7fc07c065ea0 msgr2=0x7fc07c1431c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:23.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.993+0000 7fc082234640 1 -- 172.21.15.3:0/3261684186 shutdown_connections 2023-10-30T12:10:23.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.993+0000 7fc082234640 1 -- 172.21.15.3:0/3261684186 wait complete. 2023-10-30T12:10:23.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:23.993+0000 7fc082234640 1 librados: shutdown 2023-10-30T12:10:24.008 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476741 for osd.0 2023-10-30T12:10:24.009 DEBUG:teuthology.parallel:result is None 2023-10-30T12:10:24.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.039+0000 7f88a8ff9640 1 -- 172.21.15.3:0/2783882881 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f88a0058670 con 0x7f88ac1423f0 2023-10-30T12:10:24.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.039+0000 7f88a8ff9640 10 client.4395.objecter ms_dispatch 0x7f88ac000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:24.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.039+0000 7f88a8ff9640 3 client.4395.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:24.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.039+0000 7f88a8ff9640 3 client.4395.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:24.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.039+0000 7f88a8ff9640 20 client.4395.objecter dump_active .. 0 homeless 2023-10-30T12:10:24.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.039+0000 7f88a8ff9640 1 -- 172.21.15.3:0/2783882881 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f88a0057f70 con 0x7f88ac1423f0 2023-10-30T12:10:24.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.039+0000 7f88a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:24.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.039+0000 7f88a8ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:24.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.125+0000 7faed1597640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:10:24.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.125+0000 7faed1597640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:10:24.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.125+0000 7faed1597640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:24.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.125+0000 7faed1597640 1 -- 172.21.15.3:0/2514375206 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7faecc0ab6d0 con 0x7faecc1423f0 2023-10-30T12:10:24.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.129+0000 7faeb37fe640 1 -- 172.21.15.3:0/2514375206 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faec007dd40 con 0x7faecc1423f0 2023-10-30T12:10:24.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.129+0000 7faeb37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:10:24.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.129+0000 7faeb37fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:24.131 INFO:teuthology.orchestra.run.smithi003.stdout:55834574853 2023-10-30T12:10:24.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 10 librados: watch_flush enter 2023-10-30T12:10:24.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 10 librados: watch_flush exit 2023-10-30T12:10:24.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 20 client.4366.objecter shutdown clearing up homeless session... 2023-10-30T12:10:24.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 10 client.4366.objecter successfully canceled tick 2023-10-30T12:10:24.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 1 -- 172.21.15.3:0/2514375206 >> v2:172.21.15.92:6800/1750155735 conn(0x7faeb806c810 msgr2=0x7faeb806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:24.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 1 --2- 172.21.15.3:0/2514375206 >> v2:172.21.15.92:6800/1750155735 conn(0x7faeb806c810 0x7faeb806ecd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faebc001560 tx=0x7faebc007990 comp rx=0 tx=0).stop 2023-10-30T12:10:24.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 10 monclient: shutdown 2023-10-30T12:10:24.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:24.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 1 -- 172.21.15.3:0/2514375206 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc1423f0 msgr2=0x7faecc0ab190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:24.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 1 --2- 172.21.15.3:0/2514375206 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc1423f0 0x7faecc0ab190 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faec0000c00 tx=0x7faec00046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 1 -- 172.21.15.3:0/2514375206 shutdown_connections 2023-10-30T12:10:24.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 1 --2- 172.21.15.3:0/2514375206 >> v2:172.21.15.92:6800/1750155735 conn(0x7faeb806c810 0x7faeb806ecd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 1 --2- 172.21.15.3:0/2514375206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faecc0acde0 0x7faecc1d7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 1 --2- 172.21.15.3:0/2514375206 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc1423f0 0x7faecc0ab190 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 1 --2- 172.21.15.3:0/2514375206 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faecc0a8e40 0x7faecc1459b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 1 -- 172.21.15.3:0/2514375206 >> 172.21.15.3:0/2514375206 conn(0x7faecc0b7180 msgr2=0x7faecc0b40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:24.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 1 -- 172.21.15.3:0/2514375206 shutdown_connections 2023-10-30T12:10:24.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 1 -- 172.21.15.3:0/2514375206 wait complete. 2023-10-30T12:10:24.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.131+0000 7faed1597640 1 librados: shutdown 2023-10-30T12:10:24.143 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2023-10-30T12:10:24.143 DEBUG:teuthology.parallel:result is None 2023-10-30T12:10:24.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.205+0000 7f408effd640 1 -- 172.21.15.3:0/1349513910 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f409007b7f0 con 0x7f4098145190 2023-10-30T12:10:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.205+0000 7f408effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.205+0000 7f408effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:24.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.251+0000 7fd0e67fc640 1 -- 172.21.15.3:0/2921881203 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd0e805c700 con 0x7fd0f0144f50 2023-10-30T12:10:24.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.251+0000 7fd0e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T12:10:24.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.251+0000 7fd0e67fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:24.253 INFO:teuthology.orchestra.run.smithi003.stdout:55834574853 2023-10-30T12:10:24.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 10 librados: watch_flush enter 2023-10-30T12:10:24.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 10 librados: watch_flush exit 2023-10-30T12:10:24.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 20 client.4295.objecter shutdown clearing up homeless session... 2023-10-30T12:10:24.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 10 client.4295.objecter successfully canceled tick 2023-10-30T12:10:24.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 1 -- 172.21.15.3:0/2921881203 >> v2:172.21.15.92:6800/1750155735 conn(0x7fd0d406c810 msgr2=0x7fd0d406ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:24.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 1 --2- 172.21.15.3:0/2921881203 >> v2:172.21.15.92:6800/1750155735 conn(0x7fd0d406c810 0x7fd0d406ecd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec035000 tx=0x7fd0ec030000 comp rx=0 tx=0).stop 2023-10-30T12:10:24.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 10 monclient: shutdown 2023-10-30T12:10:24.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:24.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 1 -- 172.21.15.3:0/2921881203 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0f0144f50 msgr2=0x7fd0f0141a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:24.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 1 --2- 172.21.15.3:0/2921881203 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0f0144f50 0x7fd0f0141a10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800be20 tx=0x7fd0e800bf10 comp rx=0 tx=0).stop 2023-10-30T12:10:24.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 1 -- 172.21.15.3:0/2921881203 shutdown_connections 2023-10-30T12:10:24.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 1 --2- 172.21.15.3:0/2921881203 >> v2:172.21.15.92:6800/1750155735 conn(0x7fd0d406c810 0x7fd0d406ecd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 1 --2- 172.21.15.3:0/2921881203 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0f0144f50 0x7fd0f0141a10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 1 --2- 172.21.15.3:0/2921881203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0f013eb50 0x7fd0f013f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 1 --2- 172.21.15.3:0/2921881203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0f00b4350 0x7fd0f00b7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 1 -- 172.21.15.3:0/2921881203 >> 172.21.15.3:0/2921881203 conn(0x7fd0f0065ea0 msgr2=0x7fd0f013c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:24.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 1 -- 172.21.15.3:0/2921881203 shutdown_connections 2023-10-30T12:10:24.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 1 -- 172.21.15.3:0/2921881203 wait complete. 2023-10-30T12:10:24.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.253+0000 7fd0f7ba9640 1 librados: shutdown 2023-10-30T12:10:24.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.261+0000 7f88b28cf640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T12:10:24.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.261+0000 7f88b28cf640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T12:10:24.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.261+0000 7f88b28cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:24.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.261+0000 7f88b28cf640 1 -- 172.21.15.3:0/2783882881 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f88ac13d950 con 0x7f88ac1423f0 2023-10-30T12:10:24.267 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.6 2023-10-30T12:10:24.268 DEBUG:teuthology.parallel:result is None 2023-10-30T12:10:24.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.268+0000 7f88a8ff9640 1 -- 172.21.15.3:0/2783882881 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f88a005c0c0 con 0x7f88ac1423f0 2023-10-30T12:10:24.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.268+0000 7f88a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T12:10:24.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.268+0000 7f88a8ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:24.270 INFO:teuthology.orchestra.run.smithi003.stdout:68719476741 2023-10-30T12:10:24.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.271+0000 7f88b28cf640 10 librados: watch_flush enter 2023-10-30T12:10:24.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.271+0000 7f88b28cf640 10 librados: watch_flush exit 2023-10-30T12:10:24.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.271+0000 7f88b28cf640 20 client.4395.objecter shutdown clearing up homeless session... 2023-10-30T12:10:24.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.271+0000 7f88b28cf640 10 client.4395.objecter successfully canceled tick 2023-10-30T12:10:24.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.271+0000 7f88b28cf640 1 -- 172.21.15.3:0/2783882881 >> v2:172.21.15.92:6800/1750155735 conn(0x7f889806c7c0 msgr2=0x7f889806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:24.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.271+0000 7f88b28cf640 1 --2- 172.21.15.3:0/2783882881 >> v2:172.21.15.92:6800/1750155735 conn(0x7f889806c7c0 0x7f889806ec80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f889c0015f0 tx=0x7f889c007a40 comp rx=0 tx=0).stop 2023-10-30T12:10:24.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.271+0000 7f88b28cf640 10 monclient: shutdown 2023-10-30T12:10:24.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.271+0000 7f88b28cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:24.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.271+0000 7f88b28cf640 1 -- 172.21.15.3:0/2783882881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac1423f0 msgr2=0x7f88ac121460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:24.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.271+0000 7f88b28cf640 1 --2- 172.21.15.3:0/2783882881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac1423f0 0x7f88ac121460 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f88a000b4a0 tx=0x7f88a000b960 comp rx=0 tx=0).stop 2023-10-30T12:10:24.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.271+0000 7f88b28cf640 1 -- 172.21.15.3:0/2783882881 shutdown_connections 2023-10-30T12:10:24.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.271+0000 7f88b28cf640 1 --2- 172.21.15.3:0/2783882881 >> v2:172.21.15.92:6800/1750155735 conn(0x7f889806c7c0 0x7f889806ec80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.271+0000 7f88b28cf640 1 --2- 172.21.15.3:0/2783882881 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88ac1dae20 0x7f88ac13d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.271+0000 7f88b28cf640 1 --2- 172.21.15.3:0/2783882881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac1423f0 0x7f88ac121460 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.271+0000 7f88b28cf640 1 --2- 172.21.15.3:0/2783882881 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f88ac0a8e40 0x7f88ac1da400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.272+0000 7f88b28cf640 1 -- 172.21.15.3:0/2783882881 >> 172.21.15.3:0/2783882881 conn(0x7f88ac0b7180 msgr2=0x7f88ac0b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:24.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.272+0000 7f88b28cf640 1 -- 172.21.15.3:0/2783882881 shutdown_connections 2023-10-30T12:10:24.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.272+0000 7f88b28cf640 1 -- 172.21.15.3:0/2783882881 wait complete. 2023-10-30T12:10:24.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.272+0000 7f88b28cf640 1 librados: shutdown 2023-10-30T12:10:24.285 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476741 for osd.3 2023-10-30T12:10:24.286 DEBUG:teuthology.parallel:result is None 2023-10-30T12:10:24.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.406+0000 7f409cc7c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:10:24.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.406+0000 7f409cc7c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:10:24.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.406+0000 7f409cc7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:10:24.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.406+0000 7f409cc7c640 1 -- 172.21.15.3:0/1349513910 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f40980aee80 con 0x7f4098145190 2023-10-30T12:10:24.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.412+0000 7f408effd640 1 -- 172.21.15.3:0/1349513910 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f409007b0f0 con 0x7f4098145190 2023-10-30T12:10:24.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.412+0000 7f408effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:10:24.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.412+0000 7f408effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:24.414 INFO:teuthology.orchestra.run.smithi003.stdout:55834574853 2023-10-30T12:10:24.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 10 librados: watch_flush enter 2023-10-30T12:10:24.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 10 librados: watch_flush exit 2023-10-30T12:10:24.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 20 client.4301.objecter shutdown clearing up homeless session... 2023-10-30T12:10:24.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 10 client.4301.objecter successfully canceled tick 2023-10-30T12:10:24.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 1 -- 172.21.15.3:0/1349513910 >> v2:172.21.15.92:6800/1750155735 conn(0x7f407806c7f0 msgr2=0x7f407806ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:24.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 1 --2- 172.21.15.3:0/1349513910 >> v2:172.21.15.92:6800/1750155735 conn(0x7f407806c7f0 0x7f407806ecb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4088009710 tx=0x7f4088006c90 comp rx=0 tx=0).stop 2023-10-30T12:10:24.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 10 monclient: shutdown 2023-10-30T12:10:24.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:24.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 1 -- 172.21.15.3:0/1349513910 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4098145190 msgr2=0x7f40981d66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:24.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 1 --2- 172.21.15.3:0/1349513910 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4098145190 0x7f40981d66a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4090009e50 tx=0x7f4090004a40 comp rx=0 tx=0).stop 2023-10-30T12:10:24.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 1 -- 172.21.15.3:0/1349513910 shutdown_connections 2023-10-30T12:10:24.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 1 --2- 172.21.15.3:0/1349513910 >> v2:172.21.15.92:6800/1750155735 conn(0x7f407806c7f0 0x7f407806ecb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 1 --2- 172.21.15.3:0/1349513910 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4098145190 0x7f40981d66a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 1 --2- 172.21.15.3:0/1349513910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4098071360 0x7f40981da240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 1 --2- 172.21.15.3:0/1349513910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f409806bc40 0x7f409813f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 1 -- 172.21.15.3:0/1349513910 >> 172.21.15.3:0/1349513910 conn(0x7f4098066470 msgr2=0x7f40980b0310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:24.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 1 -- 172.21.15.3:0/1349513910 shutdown_connections 2023-10-30T12:10:24.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 1 -- 172.21.15.3:0/1349513910 wait complete. 2023-10-30T12:10:24.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.414+0000 7f409cc7c640 1 librados: shutdown 2023-10-30T12:10:24.426 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2023-10-30T12:10:24.426 DEBUG:teuthology.parallel:result is None 2023-10-30T12:10:24.426 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T12:10:24.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:10:24.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.532+0000 7fac2c251640 10 monclient: get_monmap_and_config 2023-10-30T12:10:24.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.532+0000 7fac2c251640 10 monclient: build_initial_monmap 2023-10-30T12:10:24.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.532+0000 7fac2c251640 10 monclient: monmap: 2023-10-30T12:10:24.534 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:24.534 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:24.534 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:24.533141+0000 2023-10-30T12:10:24.535 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:24.533141+0000 2023-10-30T12:10:24.535 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:24.535 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:24.537 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.532+0000 7fac2c251640 1 Processor -- start 2023-10-30T12:10:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 1 -- start start 2023-10-30T12:10:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 10 monclient: init 2023-10-30T12:10:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-10-30T12:10:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fac240b4390 0x7fac240a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 10 monclient(hunting): picked mon.noname-c con 0x7fac240b4390 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac241418a0 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fac24141ce0 0x7fac240b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 10 monclient(hunting): picked mon.noname-a con 0x7fac24141ce0 addr v2:172.21.15.3:3300/0 2023-10-30T12:10:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac240a8f60 con 0x7fac241418a0 2023-10-30T12:10:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac24067a10 con 0x7fac24141ce0 2023-10-30T12:10:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac240b4770 con 0x7fac240b4390 2023-10-30T12:10:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.533+0000 7fac2c251640 10 monclient(hunting): authenticate will time out at 1415.747749s 2023-10-30T12:10:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.534+0000 7fac29fed640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fac240b4390 0x7fac240a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.534+0000 7fac2a7ee640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fac24141ce0 0x7fac240b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.534+0000 7fac29fed640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fac240b4390 0x7fac240a8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34082/0 (socket says 172.21.15.3:34082) 2023-10-30T12:10:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.534+0000 7fac29fed640 1 -- 172.21.15.3:0/2503224722 learned_addr learned my addr 172.21.15.3:0/2503224722 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:24.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.534+0000 7fac29fed640 10 monclient(hunting): get_auth_request con 0x7fac240b4390 auth_method 0 2023-10-30T12:10:24.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.534+0000 7fac29fed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:24.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.534+0000 7fac29fed640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:24.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.534+0000 7fac29fed640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:24.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.534+0000 7fac2a7ee640 10 monclient(hunting): get_auth_request con 0x7fac24141ce0 auth_method 0 2023-10-30T12:10:24.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.534+0000 7fac2a7ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:24.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.534+0000 7fac2a7ee640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:24.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.534+0000 7fac2a7ee640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:24.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.535+0000 7fac2a7ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:24.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.535+0000 7fac2a7ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:24.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.535+0000 7fac2a7ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:24.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.535+0000 7fac29fed640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:24.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.535+0000 7fac29fed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:24.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.535+0000 7fac29fed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:24.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.536+0000 7fac2a7ee640 10 monclient(hunting): handle_auth_done global_id 4398 payload 274 2023-10-30T12:10:24.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.536+0000 7fac2a7ee640 1 -- 172.21.15.3:0/2503224722 >> v2:172.21.15.92:3300/0 conn(0x7fac240b4390 msgr2=0x7fac240a8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:24.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.536+0000 7fac2a7ee640 1 --2- 172.21.15.3:0/2503224722 >> v2:172.21.15.92:3300/0 conn(0x7fac240b4390 0x7fac240a8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.536+0000 7fac2a7ee640 1 -- 172.21.15.3:0/2503224722 >> v1:172.21.15.92:6789/0 conn(0x7fac241418a0 legacy=0x7fac240b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:24.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.536+0000 7fac2a7ee640 10 monclient: _finish_hunting 0 2023-10-30T12:10:24.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.536+0000 7fac2a7ee640 1 monclient: found mon.noname-a 2023-10-30T12:10:24.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.536+0000 7fac2a7ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:24.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.536+0000 7fac2a7ee640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:24.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.536+0000 7fac2a7ee640 1 -- 172.21.15.3:0/2503224722 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac240aa3d0 con 0x7fac24141ce0 2023-10-30T12:10:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.538+0000 7fac2a7ee640 1 --2- 172.21.15.3:0/2503224722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac24141ce0 0x7fac240b2210 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fac240b4a50 tx=0x7fac2002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ffe5fb02a897e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac28feb640 1 -- 172.21.15.3:0/2503224722 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2002cc80 con 0x7fac24141ce0 2023-10-30T12:10:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac28feb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac28feb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac28feb640 10 monclient: dump: 2023-10-30T12:10:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:24.554 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac28feb640 10 monclient: _finish_auth 0 2023-10-30T12:10:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac28feb640 10 monclient: _check_auth_tickets 2023-10-30T12:10:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac28feb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac28feb640 1 -- 172.21.15.3:0/2503224722 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2002cdf0 con 0x7fac24141ce0 2023-10-30T12:10:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac28feb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac28feb640 1 -- 172.21.15.3:0/2503224722 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac20034660 con 0x7fac24141ce0 2023-10-30T12:10:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac28feb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac28feb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac28feb640 10 monclient: dump: 2023-10-30T12:10:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:24.557 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:24.557 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:24.557 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:24.557 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:24.557 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:24.558 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:24.558 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:24.558 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:24.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac2c251640 5 monclient: authenticate success, global_id 4398 2023-10-30T12:10:24.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac2c251640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:24.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.542+0000 7fac2c251640 4 set_mon_vals no callback set 2023-10-30T12:10:24.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.543+0000 7fac2c251640 10 monclient: shutdown 2023-10-30T12:10:24.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.543+0000 7fac2c251640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:24.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.543+0000 7fac2c251640 1 -- 172.21.15.3:0/2503224722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac24141ce0 msgr2=0x7fac240b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:24.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.543+0000 7fac2c251640 1 --2- 172.21.15.3:0/2503224722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac24141ce0 0x7fac240b2210 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fac240b4a50 tx=0x7fac2002bed0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.543+0000 7fac2c251640 1 -- 172.21.15.3:0/2503224722 shutdown_connections 2023-10-30T12:10:24.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.543+0000 7fac2c251640 1 --2- 172.21.15.3:0/2503224722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac24141ce0 0x7fac240b2210 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.543+0000 7fac2c251640 1 --2- 172.21.15.3:0/2503224722 >> v2:172.21.15.92:3300/0 conn(0x7fac240b4390 0x7fac240a8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.543+0000 7fac2c251640 1 -- 172.21.15.3:0/2503224722 >> 172.21.15.3:0/2503224722 conn(0x7fac24065ea0 msgr2=0x7fac240662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:24.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.543+0000 7fac2c251640 1 -- 172.21.15.3:0/2503224722 shutdown_connections 2023-10-30T12:10:24.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.543+0000 7fac2c251640 1 -- 172.21.15.3:0/2503224722 wait complete. 2023-10-30T12:10:24.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.544+0000 7fac2c251640 10 monclient: build_initial_monmap 2023-10-30T12:10:24.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.544+0000 7fac2c251640 10 monclient: monmap: 2023-10-30T12:10:24.561 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:24.562 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:24.562 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:24.562 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:24.562 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:24.562 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:24.563 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:24.563 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:24.563 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:24.563 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:24.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.544+0000 7fac2c251640 1 librados: starting msgr at 2023-10-30T12:10:24.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.544+0000 7fac2c251640 1 librados: starting objecter 2023-10-30T12:10:24.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.544+0000 7fac2c251640 1 Processor -- start 2023-10-30T12:10:24.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.544+0000 7fac2c251640 1 -- start start 2023-10-30T12:10:24.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.544+0000 7fac2c251640 1 librados: setting wanted keys 2023-10-30T12:10:24.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.544+0000 7fac2c251640 1 librados: calling monclient init 2023-10-30T12:10:24.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.544+0000 7fac2c251640 10 monclient: init 2023-10-30T12:10:24.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:24.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:24.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac240b4390 0x7fac241447c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:24.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 10 monclient(hunting): picked mon.noname-b con 0x7fac240b4390 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:24.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac241418a0 0x7fac24144df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:24.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 10 monclient(hunting): picked mon.noname-a con 0x7fac241418a0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:24.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac24141ce0 0x7fac24136480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:24.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 10 monclient(hunting): picked mon.noname-c con 0x7fac24141ce0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:24.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:24.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac240b3010 con 0x7fac241418a0 2023-10-30T12:10:24.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:24.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac24145790 con 0x7fac240b4390 2023-10-30T12:10:24.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:24.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2413e980 con 0x7fac24141ce0 2023-10-30T12:10:24.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:24.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac2c251640 10 monclient(hunting): authenticate will time out at 1415.759387s 2023-10-30T12:10:24.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.545+0000 7fac297ec640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac241418a0 0x7fac24144df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:24.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac2a7ee640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac24141ce0 0x7fac24136480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:24.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac29fed640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac240b4390 0x7fac241447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:24.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac297ec640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac241418a0 0x7fac24144df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38180/0 (socket says 172.21.15.3:38180) 2023-10-30T12:10:24.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac297ec640 1 -- 172.21.15.3:0/3229165425 learned_addr learned my addr 172.21.15.3:0/3229165425 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:24.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac297ec640 10 monclient(hunting): get_auth_request con 0x7fac241418a0 auth_method 0 2023-10-30T12:10:24.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac297ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:24.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac297ec640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:24.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac297ec640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:24.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac2a7ee640 10 monclient(hunting): get_auth_request con 0x7fac24141ce0 auth_method 0 2023-10-30T12:10:24.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac2a7ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:24.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac2a7ee640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:24.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac2a7ee640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:24.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac29fed640 10 monclient(hunting): get_auth_request con 0x7fac240b4390 auth_method 0 2023-10-30T12:10:24.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac29fed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:24.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac29fed640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:24.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac29fed640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:24.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac297ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:24.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac297ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:24.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac297ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:24.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac2a7ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:24.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac2a7ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:24.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac2a7ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:24.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac29fed640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:24.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac29fed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:24.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.546+0000 7fac29fed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:24.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.549+0000 7fac297ec640 10 monclient(hunting): handle_auth_done global_id 4401 payload 824 2023-10-30T12:10:24.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.549+0000 7fac297ec640 1 -- 172.21.15.3:0/3229165425 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac24141ce0 msgr2=0x7fac24136480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:24.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.549+0000 7fac297ec640 1 --2- 172.21.15.3:0/3229165425 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac24141ce0 0x7fac24136480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.549+0000 7fac297ec640 1 -- 172.21.15.3:0/3229165425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac240b4390 msgr2=0x7fac241447c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:24.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.549+0000 7fac297ec640 1 --2- 172.21.15.3:0/3229165425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac240b4390 0x7fac241447c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:24.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.549+0000 7fac297ec640 10 monclient: _finish_hunting 0 2023-10-30T12:10:24.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.549+0000 7fac297ec640 1 monclient: found mon.noname-a 2023-10-30T12:10:24.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.549+0000 7fac297ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:24.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.549+0000 7fac297ec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:10:24.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.549+0000 7fac297ec640 1 -- 172.21.15.3:0/3229165425 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac24145330 con 0x7fac241418a0 2023-10-30T12:10:24.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.552+0000 7fac297ec640 1 --2- 172.21.15.3:0/3229165425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac241418a0 0x7fac24144df0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fac1400c450 tx=0x7fac1400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:24.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.552+0000 7fac1a7fc640 10 client.?.objecter ms_handle_connect 0x7fac241418a0 2023-10-30T12:10:24.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.552+0000 7fac1a7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:24.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac1a7fc640 1 -- 172.21.15.3:0/3229165425 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac140052d0 con 0x7fac241418a0 2023-10-30T12:10:24.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:24.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac1a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:24.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac1a7fc640 10 monclient: dump: 2023-10-30T12:10:24.579 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:24.580 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:24.580 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:24.580 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:24.580 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:24.581 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:24.581 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:24.581 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:24.581 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:24.581 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:24.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac1a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:10:24.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac1a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:10:24.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:24.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac1a7fc640 1 -- 172.21.15.3:0/3229165425 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1400e040 con 0x7fac241418a0 2023-10-30T12:10:24.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac2c251640 5 monclient: authenticate success, global_id 4401 2023-10-30T12:10:24.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac2c251640 10 monclient: _renew_subs 2023-10-30T12:10:24.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac2c251640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:24.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac2c251640 1 -- 172.21.15.3:0/3229165425 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac241369c0 con 0x7fac241418a0 2023-10-30T12:10:24.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac1a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:24.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac1a7fc640 1 -- 172.21.15.3:0/3229165425 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac14012630 con 0x7fac241418a0 2023-10-30T12:10:24.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:24.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac1a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:24.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac1a7fc640 10 monclient: dump: 2023-10-30T12:10:24.584 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:24.584 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:24.586 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:24.586 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:24.586 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:24.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac1bfff640 4 set_mon_vals no callback set 2023-10-30T12:10:24.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac2c251640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac2c251640 10 monclient: _renew_subs 2023-10-30T12:10:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac2c251640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac2c251640 1 -- 172.21.15.3:0/3229165425 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac24136ee0 con 0x7fac241418a0 2023-10-30T12:10:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.555+0000 7fac2c251640 1 librados: init done 2023-10-30T12:10:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.557+0000 7fac2c251640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:24.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.557+0000 7fac2c251640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:24.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.557+0000 7fac2c251640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:10:24.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.557+0000 7fac2c251640 1 -- 172.21.15.3:0/3229165425 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac240a9900 con 0x7fac241418a0 2023-10-30T12:10:24.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.576+0000 7fac1a7fc640 1 -- 172.21.15.3:0/3229165425 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fac140127a0 con 0x7fac241418a0 2023-10-30T12:10:24.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.577+0000 7fac1a7fc640 1 --2- 172.21.15.3:0/3229165425 >> v2:172.21.15.92:6800/1750155735 conn(0x7fac0806c840 0x7fac0806ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.577+0000 7fac29fed640 1 --2- 172.21.15.3:0/3229165425 >> v2:172.21.15.92:6800/1750155735 conn(0x7fac0806c840 0x7fac0806ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.577+0000 7fac29fed640 10 monclient: get_auth_request con 0x7fac0806c840 auth_method 0 2023-10-30T12:10:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.578+0000 7fac29fed640 1 --2- 172.21.15.3:0/3229165425 >> v2:172.21.15.92:6800/1750155735 conn(0x7fac0806c840 0x7fac0806ed00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009bd0 tx=0x7fac1c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:24.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.578+0000 7fac1a7fc640 10 client.4401.objecter ms_handle_connect 0x7fac0806c840 2023-10-30T12:10:24.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.579+0000 7fac1a7fc640 1 -- 172.21.15.3:0/3229165425 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fac140582a0 con 0x7fac241418a0 2023-10-30T12:10:24.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.579+0000 7fac1a7fc640 10 client.4401.objecter ms_dispatch 0x7fac24000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:24.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.579+0000 7fac1a7fc640 3 client.4401.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:24.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.579+0000 7fac1a7fc640 3 client.4401.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:24.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.580+0000 7fac1a7fc640 20 client.4401.objecter dump_active .. 0 homeless 2023-10-30T12:10:24.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.837+0000 7fac1a7fc640 1 -- 172.21.15.3:0/3229165425 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fac1405c340 con 0x7fac241418a0 2023-10-30T12:10:24.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.837+0000 7fac1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:24.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:24.837+0000 7fac1a7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:25.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.018+0000 7fac2c251640 1 -- 172.21.15.3:0/3229165425 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fac24142110 con 0x7fac0806c840 2023-10-30T12:10:25.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.020+0000 7fac1a7fc640 1 -- 172.21.15.3:0/3229165425 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48056 (secure 0 0 0) 0x7fac1c009900 con 0x7fac0806c840 2023-10-30T12:10:25.022 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:10:25.023 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-10-30T12:10:25.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.022+0000 7fac2c251640 10 librados: watch_flush enter 2023-10-30T12:10:25.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.022+0000 7fac2c251640 10 librados: watch_flush exit 2023-10-30T12:10:25.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.022+0000 7fac2c251640 20 client.4401.objecter shutdown clearing up homeless session... 2023-10-30T12:10:25.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.022+0000 7fac2c251640 10 client.4401.objecter successfully canceled tick 2023-10-30T12:10:25.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.022+0000 7fac2c251640 1 -- 172.21.15.3:0/3229165425 >> v2:172.21.15.92:6800/1750155735 conn(0x7fac0806c840 msgr2=0x7fac0806ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:25.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.022+0000 7fac2c251640 1 --2- 172.21.15.3:0/3229165425 >> v2:172.21.15.92:6800/1750155735 conn(0x7fac0806c840 0x7fac0806ed00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009bd0 tx=0x7fac1c009240 comp rx=0 tx=0).stop 2023-10-30T12:10:25.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.022+0000 7fac2c251640 10 monclient: shutdown 2023-10-30T12:10:25.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.022+0000 7fac2c251640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:25.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.022+0000 7fac2c251640 1 -- 172.21.15.3:0/3229165425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac241418a0 msgr2=0x7fac24144df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:25.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.022+0000 7fac2c251640 1 --2- 172.21.15.3:0/3229165425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac241418a0 0x7fac24144df0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fac1400c450 tx=0x7fac1400c910 comp rx=0 tx=0).stop 2023-10-30T12:10:25.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.023+0000 7fac2c251640 1 -- 172.21.15.3:0/3229165425 shutdown_connections 2023-10-30T12:10:25.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.023+0000 7fac2c251640 1 --2- 172.21.15.3:0/3229165425 >> v2:172.21.15.92:6800/1750155735 conn(0x7fac0806c840 0x7fac0806ed00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.023+0000 7fac2c251640 1 --2- 172.21.15.3:0/3229165425 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac24141ce0 0x7fac24136480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.023+0000 7fac2c251640 1 --2- 172.21.15.3:0/3229165425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac241418a0 0x7fac24144df0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.023+0000 7fac2c251640 1 --2- 172.21.15.3:0/3229165425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac240b4390 0x7fac241447c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.023+0000 7fac2c251640 1 -- 172.21.15.3:0/3229165425 >> 172.21.15.3:0/3229165425 conn(0x7fac24065ea0 msgr2=0x7fac2413bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:25.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.023+0000 7fac2c251640 1 -- 172.21.15.3:0/3229165425 shutdown_connections 2023-10-30T12:10:25.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.023+0000 7fac2c251640 1 -- 172.21.15.3:0/3229165425 wait complete. 2023-10-30T12:10:25.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.023+0000 7fac2c251640 1 librados: shutdown 2023-10-30T12:10:25.035 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-10-30T12:10:23.769927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217720,"kb_used_data":1516,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757000,"statfs":{"total":773094113280,"available":772871168000,"internally_reserved":0,"allocated":1552384,"data_stored":1303935,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.283500"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:10:19.035397+0000","last_change":"2023-10-30T12:10:17.745135+0000","last_active":"2023-10-30T12:10:19.035397+0000","last_peered":"2023-10-30T12:10:19.035397+0000","last_clean":"2023-10-30T12:10:19.035397+0000","last_became_active":"2023-10-30T12:10:17.738241+0000","last_became_peered":"2023-10-30T12:10:17.738241+0000","last_unstale":"2023-10-30T12:10:19.035397+0000","last_undegraded":"2023-10-30T12:10:19.035397+0000","last_fullsized":"2023-10-30T12:10:19.035397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:01.788933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:10:20.094810+0000","last_change":"2023-10-30T12:10:17.790588+0000","last_active":"2023-10-30T12:10:20.094810+0000","last_peered":"2023-10-30T12:10:20.094810+0000","last_clean":"2023-10-30T12:10:20.094810+0000","last_became_active":"2023-10-30T12:10:17.763227+0000","last_became_peered":"2023-10-30T12:10:17.763227+0000","last_unstale":"2023-10-30T12:10:20.094810+0000","last_undegraded":"2023-10-30T12:10:20.094810+0000","last_fullsized":"2023-10-30T12:10:20.094810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:40:26.455087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:10:19.032659+0000","last_change":"2023-10-30T12:10:16.912669+0000","last_active":"2023-10-30T12:10:19.032659+0000","last_peered":"2023-10-30T12:10:19.032659+0000","last_clean":"2023-10-30T12:10:19.032659+0000","last_became_active":"2023-10-30T12:10:16.907205+0000","last_became_peered":"2023-10-30T12:10:16.907205+0000","last_unstale":"2023-10-30T12:10:19.032659+0000","last_undegraded":"2023-10-30T12:10:19.032659+0000","last_fullsized":"2023-10-30T12:10:19.032659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:48.591409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:10:18.473879+0000","last_change":"2023-10-30T12:10:17.613569+0000","last_active":"2023-10-30T12:10:18.473879+0000","last_peered":"2023-10-30T12:10:18.473879+0000","last_clean":"2023-10-30T12:10:18.473879+0000","last_became_active":"2023-10-30T12:10:17.582475+0000","last_became_peered":"2023-10-30T12:10:17.582475+0000","last_unstale":"2023-10-30T12:10:18.473879+0000","last_undegraded":"2023-10-30T12:10:18.473879+0000","last_fullsized":"2023-10-30T12:10:18.473879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:51.841448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:10:19.024265+0000","last_change":"2023-10-30T12:10:17.562627+0000","last_active":"2023-10-30T12:10:19.024265+0000","last_peered":"2023-10-30T12:10:19.024265+0000","last_clean":"2023-10-30T12:10:19.024265+0000","last_became_active":"2023-10-30T12:10:17.553523+0000","last_became_peered":"2023-10-30T12:10:17.553523+0000","last_unstale":"2023-10-30T12:10:19.024265+0000","last_undegraded":"2023-10-30T12:10:19.024265+0000","last_fullsized":"2023-10-30T12:10:19.024265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:05:16.522019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:10:20.727372+0000","last_change":"2023-10-30T12:10:16.841521+0000","last_active":"2023-10-30T12:10:20.727372+0000","last_peered":"2023-10-30T12:10:20.727372+0000","last_clean":"2023-10-30T12:10:20.727372+0000","last_became_active":"2023-10-30T12:10:16.811711+0000","last_became_peered":"2023-10-30T12:10:16.811711+0000","last_unstale":"2023-10-30T12:10:20.727372+0000","last_undegraded":"2023-10-30T12:10:20.727372+0000","last_fullsized":"2023-10-30T12:10:20.727372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:57.059004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:10:19.018289+0000","last_change":"2023-10-30T12:10:16.879470+0000","last_active":"2023-10-30T12:10:19.018289+0000","last_peered":"2023-10-30T12:10:19.018289+0000","last_clean":"2023-10-30T12:10:19.018289+0000","last_became_active":"2023-10-30T12:10:16.849940+0000","last_became_peered":"2023-10-30T12:10:16.849940+0000","last_unstale":"2023-10-30T12:10:19.018289+0000","last_undegraded":"2023-10-30T12:10:19.018289+0000","last_fullsized":"2023-10-30T12:10:19.018289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:09:07.690364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:10:19.027292+0000","last_change":"2023-10-30T12:10:17.161654+0000","last_active":"2023-10-30T12:10:19.027292+0000","last_peered":"2023-10-30T12:10:19.027292+0000","last_clean":"2023-10-30T12:10:19.027292+0000","last_became_active":"2023-10-30T12:10:17.153120+0000","last_became_peered":"2023-10-30T12:10:17.153120+0000","last_unstale":"2023-10-30T12:10:19.027292+0000","last_undegraded":"2023-10-30T12:10:19.027292+0000","last_fullsized":"2023-10-30T12:10:19.027292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:42.496061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":119,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:10:18.514779+0000","last_change":"2023-10-30T12:10:09.781745+0000","last_active":"2023-10-30T12:10:18.514779+0000","last_peered":"2023-10-30T12:10:18.514779+0000","last_clean":"2023-10-30T12:10:18.514779+0000","last_became_active":"2023-10-30T12:10:09.748373+0000","last_became_peered":"2023-10-30T12:10:09.748373+0000","last_unstale":"2023-10-30T12:10:18.514779+0000","last_undegraded":"2023-10-30T12:10:18.514779+0000","last_fullsized":"2023-10-30T12:10:18.514779+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:07.000545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:07.000545+0000","last_clean_scrub_stamp":"2023-10-30T12:10:07.000545+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:30.943478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476742,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":65351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.473000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.872}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.797000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.524999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4920000000000009}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.832000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.841999999999999}]}]},{"osd":2,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":60723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.538}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.768999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.079000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.809999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.581}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.716000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.396999999999998}]}]},{"osd":0,"up_from":16,"seq":68719476741,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":65351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":54.273000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":39.838999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.346}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":41.442}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":44.131999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":51.799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":47.762}]}]},{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":65351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.257}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.923}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6020000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.858000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.15}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":54.872999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.103000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":463191,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.634}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":98.125}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.451000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":100.146}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":86.518000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":101.2}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":102.761}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":60723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6059999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.192}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.369}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.800999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.538}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.215}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.358000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":457894,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.069000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.847999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.974}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.173}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.169}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.577000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.413}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":65351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.5190000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.751000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.445}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.611000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6999999999999993}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.227}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.512}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:10:25.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:10:25.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.141+0000 7fbe3a7bb640 10 monclient: get_monmap_and_config 2023-10-30T12:10:25.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.141+0000 7fbe3a7bb640 10 monclient: build_initial_monmap 2023-10-30T12:10:25.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.141+0000 7fbe3a7bb640 10 monclient: monmap: 2023-10-30T12:10:25.143 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:25.143 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:25.144 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:25.142614+0000 2023-10-30T12:10:25.144 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:25.142614+0000 2023-10-30T12:10:25.144 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:25.144 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:25.145 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:25.145 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:25.145 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:25.145 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:25.146 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:25.146 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:25.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.142+0000 7fbe3a7bb640 1 Processor -- start 2023-10-30T12:10:25.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.142+0000 7fbe3a7bb640 1 -- start start 2023-10-30T12:10:25.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.142+0000 7fbe3a7bb640 10 monclient: init 2023-10-30T12:10:25.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.142+0000 7fbe3a7bb640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:25.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.142+0000 7fbe3a7bb640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-10-30T12:10:25.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.142+0000 7fbe3a7bb640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbe340b4390 0x7fbe3413ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:25.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.142+0000 7fbe3a7bb640 10 monclient(hunting): picked mon.noname-c con 0x7fbe340b4390 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:25.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.142+0000 7fbe3a7bb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe340b12d0 addr v1:172.21.15.92:6789/0 2023-10-30T12:10:25.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.142+0000 7fbe3a7bb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbe34141910 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:25.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.142+0000 7fbe3a7bb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe341424b0 con 0x7fbe34141910 2023-10-30T12:10:25.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.143+0000 7fbe3a7bb640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe340aa440 con 0x7fbe340b12d0 2023-10-30T12:10:25.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.143+0000 7fbe3a7bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:25.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.143+0000 7fbe3a7bb640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe34067a10 con 0x7fbe340b4390 2023-10-30T12:10:25.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.143+0000 7fbe3a7bb640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:25.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.143+0000 7fbe3a7bb640 10 monclient(hunting): authenticate will time out at 1416.357082s 2023-10-30T12:10:25.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.143+0000 7fbe38d58640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbe34141910 0x7fbe34141da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:39622/0 (socket says 172.21.15.3:39622) 2023-10-30T12:10:25.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.143+0000 7fbe38d58640 1 -- 172.21.15.3:0/951368343 learned_addr learned my addr 172.21.15.3:0/951368343 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:25.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.143+0000 7fbe33fff640 1 --2- 172.21.15.3:0/951368343 >> v2:172.21.15.92:3300/0 conn(0x7fbe340b4390 0x7fbe3413ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:25.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.144+0000 7fbe33fff640 10 monclient(hunting): get_auth_request con 0x7fbe340b4390 auth_method 0 2023-10-30T12:10:25.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.144+0000 7fbe33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:25.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.144+0000 7fbe33fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:25.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.144+0000 7fbe33fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:25.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.144+0000 7fbe33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:25.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.144+0000 7fbe33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:25.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.144+0000 7fbe33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:25.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.146+0000 7fbe32ffd640 1 -- 172.21.15.3:0/951368343 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886149738 0 0) 0x7fbe2c001490 con 0x7fbe34141910 2023-10-30T12:10:25.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.146+0000 7fbe32ffd640 10 monclient(hunting): _negotiate 2023-10-30T12:10:25.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.146+0000 7fbe32ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:25.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.146+0000 7fbe32ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:25.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.146+0000 7fbe32ffd640 10 monclient(hunting): my global_id is 4404 2023-10-30T12:10:25.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.146+0000 7fbe32ffd640 1 -- 172.21.15.3:0/951368343 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe1c003520 con 0x7fbe34141910 2023-10-30T12:10:25.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.146+0000 7fbe33fff640 10 monclient(hunting): handle_auth_done global_id 4375 payload 274 2023-10-30T12:10:25.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.146+0000 7fbe33fff640 1 -- 172.21.15.3:0/951368343 >> v1:172.21.15.92:6789/0 conn(0x7fbe340b12d0 legacy=0x7fbe3413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:25.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.146+0000 7fbe33fff640 1 -- 172.21.15.3:0/951368343 >> v1:172.21.15.3:6789/0 conn(0x7fbe34141910 legacy=0x7fbe34141da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:25.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.146+0000 7fbe33fff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:25.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.146+0000 7fbe33fff640 1 monclient: found mon.noname-c 2023-10-30T12:10:25.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.146+0000 7fbe33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:25.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.146+0000 7fbe33fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:10:25.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.146+0000 7fbe33fff640 1 -- 172.21.15.3:0/951368343 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe340a8800 con 0x7fbe340b4390 2023-10-30T12:10:25.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.149+0000 7fbe33fff640 1 --2- 172.21.15.3:0/951368343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe340b4390 0x7fbe3413ea50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbe280097a0 tx=0x7fbe2802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6891e79763b12bcd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:25.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe32ffd640 1 -- 172.21.15.3:0/951368343 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2802f810 con 0x7fbe340b4390 2023-10-30T12:10:25.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:25.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe32ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:25.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe32ffd640 10 monclient: dump: 2023-10-30T12:10:25.161 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:25.161 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:25.162 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.162 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.162 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:25.162 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:25.162 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:25.162 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:25.163 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:25.163 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:25.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe32ffd640 10 monclient: _finish_auth 0 2023-10-30T12:10:25.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe32ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:25.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:25.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe32ffd640 1 -- 172.21.15.3:0/951368343 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2802f980 con 0x7fbe340b4390 2023-10-30T12:10:25.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe3a7bb640 5 monclient: authenticate success, global_id 4375 2023-10-30T12:10:25.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe3a7bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:25.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe32ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:25.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe3a7bb640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe3a7bb640 4 set_mon_vals no callback set 2023-10-30T12:10:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe32ffd640 1 -- 172.21.15.3:0/951368343 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2802fd20 con 0x7fbe340b4390 2023-10-30T12:10:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.153+0000 7fbe32ffd640 10 monclient: dump: 2023-10-30T12:10:25.165 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:25.166 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:25.166 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.166 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.166 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:25.167 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:25.167 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:25.167 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:25.167 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:25.167 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:25.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.154+0000 7fbe3a7bb640 10 monclient: shutdown 2023-10-30T12:10:25.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.154+0000 7fbe3a7bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:25.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.154+0000 7fbe3a7bb640 1 -- 172.21.15.3:0/951368343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe340b4390 msgr2=0x7fbe3413ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:25.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.154+0000 7fbe3a7bb640 1 --2- 172.21.15.3:0/951368343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe340b4390 0x7fbe3413ea50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbe280097a0 tx=0x7fbe2802b9f0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.154+0000 7fbe3a7bb640 1 -- 172.21.15.3:0/951368343 shutdown_connections 2023-10-30T12:10:25.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.154+0000 7fbe3a7bb640 1 --2- 172.21.15.3:0/951368343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe340b4390 0x7fbe3413ea50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.154+0000 7fbe3a7bb640 1 -- 172.21.15.3:0/951368343 >> 172.21.15.3:0/951368343 conn(0x7fbe34065ed0 msgr2=0x7fbe340662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:25.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.154+0000 7fbe3a7bb640 1 -- 172.21.15.3:0/951368343 shutdown_connections 2023-10-30T12:10:25.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.154+0000 7fbe3a7bb640 1 -- 172.21.15.3:0/951368343 wait complete. 2023-10-30T12:10:25.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.155+0000 7fbe3a7bb640 10 monclient: build_initial_monmap 2023-10-30T12:10:25.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.155+0000 7fbe3a7bb640 10 monclient: monmap: 2023-10-30T12:10:25.170 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:25.170 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:25.170 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:25.170 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:25.170 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:25.171 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:25.171 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:25.171 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:25.171 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:25.171 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:25.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.155+0000 7fbe3a7bb640 1 librados: starting msgr at 2023-10-30T12:10:25.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.155+0000 7fbe3a7bb640 1 librados: starting objecter 2023-10-30T12:10:25.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.155+0000 7fbe3a7bb640 1 Processor -- start 2023-10-30T12:10:25.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.155+0000 7fbe3a7bb640 1 -- start start 2023-10-30T12:10:25.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.155+0000 7fbe3a7bb640 1 librados: setting wanted keys 2023-10-30T12:10:25.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.155+0000 7fbe3a7bb640 1 librados: calling monclient init 2023-10-30T12:10:25.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.155+0000 7fbe3a7bb640 10 monclient: init 2023-10-30T12:10:25.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:25.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:10:25.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe340b12d0 0x7fbe3413bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:25.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 10 monclient(hunting): picked mon.noname-c con 0x7fbe340b12d0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:25.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe340b4390 0x7fbe341349e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:25.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 10 monclient(hunting): picked mon.noname-b con 0x7fbe340b4390 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:25.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe34141910 0x7fbe34136be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:25.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 10 monclient(hunting): picked mon.noname-a con 0x7fbe34141910 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:25.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:25.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe340a8db0 con 0x7fbe34141910 2023-10-30T12:10:25.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:25.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe34136090 con 0x7fbe340b4390 2023-10-30T12:10:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe34136700 con 0x7fbe340b12d0 2023-10-30T12:10:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe3a7bb640 10 monclient(hunting): authenticate will time out at 1416.370451s 2023-10-30T12:10:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe38d58640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe34141910 0x7fbe34136be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe33fff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe340b12d0 0x7fbe3413bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:25.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.156+0000 7fbe337fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe340b4390 0x7fbe341349e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:25.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe38d58640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe34141910 0x7fbe34136be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38190/0 (socket says 172.21.15.3:38190) 2023-10-30T12:10:25.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe38d58640 1 -- 172.21.15.3:0/1680656491 learned_addr learned my addr 172.21.15.3:0/1680656491 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:25.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe38d58640 10 monclient(hunting): get_auth_request con 0x7fbe34141910 auth_method 0 2023-10-30T12:10:25.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe38d58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:25.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe38d58640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:25.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe38d58640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:25.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe337fe640 10 monclient(hunting): get_auth_request con 0x7fbe340b4390 auth_method 0 2023-10-30T12:10:25.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:25.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe337fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:25.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe337fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:25.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe33fff640 10 monclient(hunting): get_auth_request con 0x7fbe340b12d0 auth_method 0 2023-10-30T12:10:25.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:25.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe33fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:25.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe33fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:25.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:25.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:25.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:25.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe38d58640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:25.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe38d58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:25.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe38d58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:25.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:25.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:25.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.157+0000 7fbe33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:25.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.160+0000 7fbe337fe640 10 monclient(hunting): handle_auth_done global_id 4378 payload 824 2023-10-30T12:10:25.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.160+0000 7fbe337fe640 1 -- 172.21.15.3:0/1680656491 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe340b12d0 msgr2=0x7fbe3413bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:25.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.160+0000 7fbe337fe640 1 --2- 172.21.15.3:0/1680656491 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe340b12d0 0x7fbe3413bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.160+0000 7fbe337fe640 1 -- 172.21.15.3:0/1680656491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe34141910 msgr2=0x7fbe34136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:25.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.160+0000 7fbe337fe640 1 --2- 172.21.15.3:0/1680656491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe34141910 0x7fbe34136be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.160+0000 7fbe337fe640 10 monclient: _finish_hunting 0 2023-10-30T12:10:25.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.160+0000 7fbe337fe640 1 monclient: found mon.noname-b 2023-10-30T12:10:25.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.160+0000 7fbe337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:25.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.160+0000 7fbe337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:25.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.160+0000 7fbe337fe640 1 -- 172.21.15.3:0/1680656491 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3413ac80 con 0x7fbe340b4390 2023-10-30T12:10:25.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.162+0000 7fbe337fe640 1 --2- 172.21.15.3:0/1680656491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe340b4390 0x7fbe341349e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400c340 tx=0x7fbe2400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:25.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.162+0000 7fbe30ff9640 10 client.?.objecter ms_handle_connect 0x7fbe340b4390 2023-10-30T12:10:25.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.162+0000 7fbe30ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:25.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe30ff9640 1 -- 172.21.15.3:0/1680656491 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe240052d0 con 0x7fbe340b4390 2023-10-30T12:10:25.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:25.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe30ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:25.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe30ff9640 10 monclient: dump: 2023-10-30T12:10:25.189 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:25.189 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:25.189 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.190 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.190 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:25.190 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:25.190 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:25.190 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:25.191 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:25.191 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:25.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe30ff9640 10 monclient: _finish_auth 0 2023-10-30T12:10:25.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe30ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:10:25.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:25.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe30ff9640 1 -- 172.21.15.3:0/1680656491 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2400e040 con 0x7fbe340b4390 2023-10-30T12:10:25.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe30ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:25.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe30ff9640 1 -- 172.21.15.3:0/1680656491 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe24007260 con 0x7fbe340b4390 2023-10-30T12:10:25.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:25.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe30ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:25.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe30ff9640 10 monclient: dump: 2023-10-30T12:10:25.193 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:25.193 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:25.193 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.193 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.193 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:25.194 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:25.194 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:25.194 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:25.194 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:25.194 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:25.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe3a7bb640 5 monclient: authenticate success, global_id 4378 2023-10-30T12:10:25.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe3a7bb640 10 monclient: _renew_subs 2023-10-30T12:10:25.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe3a7bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:25.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe3a7bb640 1 -- 172.21.15.3:0/1680656491 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe34134f20 con 0x7fbe340b4390 2023-10-30T12:10:25.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe327fc640 4 set_mon_vals no callback set 2023-10-30T12:10:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe3a7bb640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe3a7bb640 10 monclient: _renew_subs 2023-10-30T12:10:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe3a7bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe3a7bb640 1 -- 172.21.15.3:0/1680656491 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe34137300 con 0x7fbe340b4390 2023-10-30T12:10:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.166+0000 7fbe3a7bb640 1 librados: init done 2023-10-30T12:10:25.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.167+0000 7fbe3a7bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:25.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.167+0000 7fbe3a7bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:25.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.167+0000 7fbe3a7bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:25.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.167+0000 7fbe3a7bb640 1 -- 172.21.15.3:0/1680656491 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe34136900 con 0x7fbe340b4390 2023-10-30T12:10:25.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.186+0000 7fbe30ff9640 1 -- 172.21.15.3:0/1680656491 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fbe240073d0 con 0x7fbe340b4390 2023-10-30T12:10:25.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.186+0000 7fbe30ff9640 1 --2- 172.21.15.3:0/1680656491 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbe2006c810 0x7fbe2006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:25.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.186+0000 7fbe33fff640 1 --2- 172.21.15.3:0/1680656491 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbe2006c810 0x7fbe2006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:25.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.186+0000 7fbe33fff640 10 monclient: get_auth_request con 0x7fbe2006c810 auth_method 0 2023-10-30T12:10:25.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.187+0000 7fbe33fff640 1 --2- 172.21.15.3:0/1680656491 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbe2006c810 0x7fbe2006ecd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbe28036000 tx=0x7fbe28008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:25.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.187+0000 7fbe30ff9640 10 client.4378.objecter ms_handle_connect 0x7fbe2006c810 2023-10-30T12:10:25.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.407+0000 7fbe30ff9640 1 -- 172.21.15.3:0/1680656491 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7fbe240574b0 con 0x7fbe340b4390 2023-10-30T12:10:25.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.407+0000 7fbe30ff9640 10 client.4378.objecter ms_dispatch 0x7fbe34000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:25.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.407+0000 7fbe30ff9640 3 client.4378.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:25.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.407+0000 7fbe30ff9640 3 client.4378.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:25.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.407+0000 7fbe30ff9640 20 client.4378.objecter dump_active .. 0 homeless 2023-10-30T12:10:25.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.419+0000 7fbe30ff9640 1 -- 172.21.15.3:0/1680656491 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbe2405b550 con 0x7fbe340b4390 2023-10-30T12:10:25.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.419+0000 7fbe30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:25.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.419+0000 7fbe30ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:25.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.596+0000 7fbe3a7bb640 1 -- 172.21.15.3:0/1680656491 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbe340b4770 con 0x7fbe2006c810 2023-10-30T12:10:25.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.598+0000 7fbe30ff9640 1 -- 172.21.15.3:0/1680656491 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48056 (secure 0 0 0) 0x7fbe2802f7f0 con 0x7fbe2006c810 2023-10-30T12:10:25.600 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:10:25.601 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-10-30T12:10:25.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 10 librados: watch_flush enter 2023-10-30T12:10:25.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 10 librados: watch_flush exit 2023-10-30T12:10:25.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 20 client.4378.objecter shutdown clearing up homeless session... 2023-10-30T12:10:25.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 10 client.4378.objecter successfully canceled tick 2023-10-30T12:10:25.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 1 -- 172.21.15.3:0/1680656491 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbe2006c810 msgr2=0x7fbe2006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:25.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 1 --2- 172.21.15.3:0/1680656491 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbe2006c810 0x7fbe2006ecd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbe28036000 tx=0x7fbe28008030 comp rx=0 tx=0).stop 2023-10-30T12:10:25.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 10 monclient: shutdown 2023-10-30T12:10:25.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:25.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 1 -- 172.21.15.3:0/1680656491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe340b4390 msgr2=0x7fbe341349e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:25.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 1 --2- 172.21.15.3:0/1680656491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe340b4390 0x7fbe341349e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400c340 tx=0x7fbe2400c800 comp rx=0 tx=0).stop 2023-10-30T12:10:25.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 1 -- 172.21.15.3:0/1680656491 shutdown_connections 2023-10-30T12:10:25.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 1 --2- 172.21.15.3:0/1680656491 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbe2006c810 0x7fbe2006ecd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 1 --2- 172.21.15.3:0/1680656491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe34141910 0x7fbe34136be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 1 --2- 172.21.15.3:0/1680656491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe340b4390 0x7fbe341349e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 1 --2- 172.21.15.3:0/1680656491 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe340b12d0 0x7fbe3413bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 1 -- 172.21.15.3:0/1680656491 >> 172.21.15.3:0/1680656491 conn(0x7fbe34065ed0 msgr2=0x7fbe340b7160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 1 -- 172.21.15.3:0/1680656491 shutdown_connections 2023-10-30T12:10:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 1 -- 172.21.15.3:0/1680656491 wait complete. 2023-10-30T12:10:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.600+0000 7fbe3a7bb640 1 librados: shutdown 2023-10-30T12:10:25.612 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-10-30T12:10:23.769927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217720,"kb_used_data":1516,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757000,"statfs":{"total":773094113280,"available":772871168000,"internally_reserved":0,"allocated":1552384,"data_stored":1303935,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.283500"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:10:19.035397+0000","last_change":"2023-10-30T12:10:17.745135+0000","last_active":"2023-10-30T12:10:19.035397+0000","last_peered":"2023-10-30T12:10:19.035397+0000","last_clean":"2023-10-30T12:10:19.035397+0000","last_became_active":"2023-10-30T12:10:17.738241+0000","last_became_peered":"2023-10-30T12:10:17.738241+0000","last_unstale":"2023-10-30T12:10:19.035397+0000","last_undegraded":"2023-10-30T12:10:19.035397+0000","last_fullsized":"2023-10-30T12:10:19.035397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:01.788933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:10:20.094810+0000","last_change":"2023-10-30T12:10:17.790588+0000","last_active":"2023-10-30T12:10:20.094810+0000","last_peered":"2023-10-30T12:10:20.094810+0000","last_clean":"2023-10-30T12:10:20.094810+0000","last_became_active":"2023-10-30T12:10:17.763227+0000","last_became_peered":"2023-10-30T12:10:17.763227+0000","last_unstale":"2023-10-30T12:10:20.094810+0000","last_undegraded":"2023-10-30T12:10:20.094810+0000","last_fullsized":"2023-10-30T12:10:20.094810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:40:26.455087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:10:19.032659+0000","last_change":"2023-10-30T12:10:16.912669+0000","last_active":"2023-10-30T12:10:19.032659+0000","last_peered":"2023-10-30T12:10:19.032659+0000","last_clean":"2023-10-30T12:10:19.032659+0000","last_became_active":"2023-10-30T12:10:16.907205+0000","last_became_peered":"2023-10-30T12:10:16.907205+0000","last_unstale":"2023-10-30T12:10:19.032659+0000","last_undegraded":"2023-10-30T12:10:19.032659+0000","last_fullsized":"2023-10-30T12:10:19.032659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:48.591409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:10:18.473879+0000","last_change":"2023-10-30T12:10:17.613569+0000","last_active":"2023-10-30T12:10:18.473879+0000","last_peered":"2023-10-30T12:10:18.473879+0000","last_clean":"2023-10-30T12:10:18.473879+0000","last_became_active":"2023-10-30T12:10:17.582475+0000","last_became_peered":"2023-10-30T12:10:17.582475+0000","last_unstale":"2023-10-30T12:10:18.473879+0000","last_undegraded":"2023-10-30T12:10:18.473879+0000","last_fullsized":"2023-10-30T12:10:18.473879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:51.841448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:10:19.024265+0000","last_change":"2023-10-30T12:10:17.562627+0000","last_active":"2023-10-30T12:10:19.024265+0000","last_peered":"2023-10-30T12:10:19.024265+0000","last_clean":"2023-10-30T12:10:19.024265+0000","last_became_active":"2023-10-30T12:10:17.553523+0000","last_became_peered":"2023-10-30T12:10:17.553523+0000","last_unstale":"2023-10-30T12:10:19.024265+0000","last_undegraded":"2023-10-30T12:10:19.024265+0000","last_fullsized":"2023-10-30T12:10:19.024265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:05:16.522019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:10:20.727372+0000","last_change":"2023-10-30T12:10:16.841521+0000","last_active":"2023-10-30T12:10:20.727372+0000","last_peered":"2023-10-30T12:10:20.727372+0000","last_clean":"2023-10-30T12:10:20.727372+0000","last_became_active":"2023-10-30T12:10:16.811711+0000","last_became_peered":"2023-10-30T12:10:16.811711+0000","last_unstale":"2023-10-30T12:10:20.727372+0000","last_undegraded":"2023-10-30T12:10:20.727372+0000","last_fullsized":"2023-10-30T12:10:20.727372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:57.059004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:10:19.018289+0000","last_change":"2023-10-30T12:10:16.879470+0000","last_active":"2023-10-30T12:10:19.018289+0000","last_peered":"2023-10-30T12:10:19.018289+0000","last_clean":"2023-10-30T12:10:19.018289+0000","last_became_active":"2023-10-30T12:10:16.849940+0000","last_became_peered":"2023-10-30T12:10:16.849940+0000","last_unstale":"2023-10-30T12:10:19.018289+0000","last_undegraded":"2023-10-30T12:10:19.018289+0000","last_fullsized":"2023-10-30T12:10:19.018289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:09:07.690364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:10:19.027292+0000","last_change":"2023-10-30T12:10:17.161654+0000","last_active":"2023-10-30T12:10:19.027292+0000","last_peered":"2023-10-30T12:10:19.027292+0000","last_clean":"2023-10-30T12:10:19.027292+0000","last_became_active":"2023-10-30T12:10:17.153120+0000","last_became_peered":"2023-10-30T12:10:17.153120+0000","last_unstale":"2023-10-30T12:10:19.027292+0000","last_undegraded":"2023-10-30T12:10:19.027292+0000","last_fullsized":"2023-10-30T12:10:19.027292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:42.496061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":119,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:10:18.514779+0000","last_change":"2023-10-30T12:10:09.781745+0000","last_active":"2023-10-30T12:10:18.514779+0000","last_peered":"2023-10-30T12:10:18.514779+0000","last_clean":"2023-10-30T12:10:18.514779+0000","last_became_active":"2023-10-30T12:10:09.748373+0000","last_became_peered":"2023-10-30T12:10:09.748373+0000","last_unstale":"2023-10-30T12:10:18.514779+0000","last_undegraded":"2023-10-30T12:10:18.514779+0000","last_fullsized":"2023-10-30T12:10:18.514779+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:07.000545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:07.000545+0000","last_clean_scrub_stamp":"2023-10-30T12:10:07.000545+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:30.943478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476742,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":65351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.473000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.872}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.797000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.524999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4920000000000009}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.832000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.841999999999999}]}]},{"osd":2,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":60723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.538}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.768999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.079000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.809999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.581}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.716000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.396999999999998}]}]},{"osd":0,"up_from":16,"seq":68719476741,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":65351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":54.273000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":39.838999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.346}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":41.442}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":44.131999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":51.799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":47.762}]}]},{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":65351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.257}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.923}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6020000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.858000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.15}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":54.872999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.103000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":463191,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.634}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":98.125}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.451000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":100.146}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":86.518000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":101.2}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":102.761}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":60723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6059999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.192}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.369}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.800999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.538}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.215}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.358000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":457894,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.069000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.847999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.974}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.173}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.169}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.577000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.413}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":65351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.5190000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.751000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.445}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.611000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6999999999999993}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.227}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.512}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:10:25.614 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T12:10:25.614 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-30T12:10:25.614 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-30T12:10:25.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-30T12:10:25.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.717+0000 7f60b204b640 10 monclient: get_monmap_and_config 2023-10-30T12:10:25.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.717+0000 7f60b204b640 10 monclient: build_initial_monmap 2023-10-30T12:10:25.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.718+0000 7f60b204b640 10 monclient: monmap: 2023-10-30T12:10:25.720 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:25.720 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:25.720 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:10:25.718963+0000 2023-10-30T12:10:25.720 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:10:25.718963+0000 2023-10-30T12:10:25.720 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:25.721 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:25.721 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:10:25.721 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:10:25.721 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:10:25.721 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:10:25.721 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:10:25.722 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:25.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.718+0000 7f60b204b640 1 Processor -- start 2023-10-30T12:10:25.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.718+0000 7f60b204b640 1 -- start start 2023-10-30T12:10:25.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.718+0000 7f60b204b640 10 monclient: init 2023-10-30T12:10:25.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.719+0000 7f60b204b640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:25.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.719+0000 7f60b204b640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-10-30T12:10:25.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.719+0000 7f60b204b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60ac0b4390 0x7f60ac13ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:25.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.719+0000 7f60b204b640 10 monclient(hunting): picked mon.noname-b con 0x7f60ac0b4390 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:25.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.719+0000 7f60b204b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f60ac0b12d0 0x7f60ac13ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:25.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.719+0000 7f60b204b640 10 monclient(hunting): picked mon.noname-c con 0x7f60ac0b12d0 addr v2:172.21.15.92:3300/0 2023-10-30T12:10:25.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.719+0000 7f60b204b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60ac141910 addr v1:172.21.15.3:6789/0 2023-10-30T12:10:25.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.719+0000 7f60b204b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60ac0aa440 con 0x7f60ac141910 2023-10-30T12:10:25.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.719+0000 7f60b204b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:25.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.719+0000 7f60b204b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60ac067a10 con 0x7f60ac0b12d0 2023-10-30T12:10:25.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.719+0000 7f60b204b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:25.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.719+0000 7f60b204b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac0b4770 con 0x7f60ac0b4390 2023-10-30T12:10:25.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.719+0000 7f60b204b640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:25.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.719+0000 7f60b204b640 10 monclient(hunting): authenticate will time out at 1416.933600s 2023-10-30T12:10:25.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.720+0000 7f60ab7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60ac0b4390 0x7f60ac13ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:25.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.720+0000 7f60abfff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f60ac141910 0x7f60ac141d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:39634/0 (socket says 172.21.15.3:39634) 2023-10-30T12:10:25.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.720+0000 7f60abfff640 1 -- 172.21.15.3:0/2250328040 learned_addr learned my addr 172.21.15.3:0/2250328040 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:10:25.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.720+0000 7f60aaffd640 1 --2- 172.21.15.3:0/2250328040 >> v2:172.21.15.92:3300/0 conn(0x7f60ac0b12d0 0x7f60ac13ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:25.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.720+0000 7f60ab7fe640 10 monclient(hunting): get_auth_request con 0x7f60ac0b4390 auth_method 0 2023-10-30T12:10:25.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.720+0000 7f60ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:25.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.720+0000 7f60ab7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:25.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.720+0000 7f60ab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:25.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.720+0000 7f60aaffd640 10 monclient(hunting): get_auth_request con 0x7f60ac0b12d0 auth_method 0 2023-10-30T12:10:25.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.720+0000 7f60aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:25.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.720+0000 7f60aaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:25.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.720+0000 7f60aaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:25.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.721+0000 7f60aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:25.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.721+0000 7f60aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:25.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.721+0000 7f60aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:25.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.721+0000 7f60ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:25.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.721+0000 7f60ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:25.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.721+0000 7f60ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:25.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.722+0000 7f60aaffd640 10 monclient(hunting): handle_auth_done global_id 4381 payload 274 2023-10-30T12:10:25.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.722+0000 7f60aaffd640 1 -- 172.21.15.3:0/2250328040 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60ac0b4390 msgr2=0x7f60ac13ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:25.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.722+0000 7f60aaffd640 1 --2- 172.21.15.3:0/2250328040 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60ac0b4390 0x7f60ac13ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.722+0000 7f60aaffd640 1 -- 172.21.15.3:0/2250328040 >> v1:172.21.15.3:6789/0 conn(0x7f60ac141910 legacy=0x7f60ac141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:25.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.722+0000 7f60aaffd640 10 monclient: _finish_hunting 0 2023-10-30T12:10:25.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.722+0000 7f60aaffd640 1 monclient: found mon.noname-c 2023-10-30T12:10:25.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.722+0000 7f60aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:25.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.722+0000 7f60aaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:10:25.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.722+0000 7f60aaffd640 1 -- 172.21.15.3:0/2250328040 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac0a8800 con 0x7f60ac0b12d0 2023-10-30T12:10:25.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.724+0000 7f60aaffd640 1 --2- 172.21.15.3:0/2250328040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60ac0b12d0 0x7f60ac13ef90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f60a00097a0 tx=0x7f60a002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a7d38da3c4ca8ac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:25.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60aa7fc640 1 -- 172.21.15.3:0/2250328040 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0030610 con 0x7f60ac0b12d0 2023-10-30T12:10:25.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:25.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60aa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:25.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60aa7fc640 10 monclient: dump: 2023-10-30T12:10:25.736 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:25.736 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:25.736 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.737 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.737 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:25.737 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:25.737 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:25.737 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:25.738 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:25.738 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:25.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60aa7fc640 10 monclient: _finish_auth 0 2023-10-30T12:10:25.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60aa7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:10:25.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:25.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60aa7fc640 1 -- 172.21.15.3:0/2250328040 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a0030780 con 0x7f60ac0b12d0 2023-10-30T12:10:25.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60aa7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:25.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60aa7fc640 1 -- 172.21.15.3:0/2250328040 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0030b20 con 0x7f60ac0b12d0 2023-10-30T12:10:25.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60b204b640 5 monclient: authenticate success, global_id 4381 2023-10-30T12:10:25.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60b204b640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:25.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60b204b640 4 set_mon_vals no callback set 2023-10-30T12:10:25.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:25.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:25.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60aa7fc640 10 monclient: dump: 2023-10-30T12:10:25.740 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:25.740 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:25.741 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.741 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.741 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:25.741 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:25.741 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:25.742 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:25.742 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:25.742 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:25.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60b204b640 10 monclient: shutdown 2023-10-30T12:10:25.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60b204b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:25.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60b204b640 1 -- 172.21.15.3:0/2250328040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60ac0b12d0 msgr2=0x7f60ac13ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:25.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60b204b640 1 --2- 172.21.15.3:0/2250328040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60ac0b12d0 0x7f60ac13ef90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f60a00097a0 tx=0x7f60a002b960 comp rx=0 tx=0).stop 2023-10-30T12:10:25.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60b204b640 1 -- 172.21.15.3:0/2250328040 shutdown_connections 2023-10-30T12:10:25.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60b204b640 1 --2- 172.21.15.3:0/2250328040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60ac0b12d0 0x7f60ac13ef90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60b204b640 1 --2- 172.21.15.3:0/2250328040 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60ac0b4390 0x7f60ac13ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60b204b640 1 -- 172.21.15.3:0/2250328040 >> 172.21.15.3:0/2250328040 conn(0x7f60ac065ed0 msgr2=0x7f60ac0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:25.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60b204b640 1 -- 172.21.15.3:0/2250328040 shutdown_connections 2023-10-30T12:10:25.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.728+0000 7f60b204b640 1 -- 172.21.15.3:0/2250328040 wait complete. 2023-10-30T12:10:25.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.729+0000 7f60b204b640 10 monclient: build_initial_monmap 2023-10-30T12:10:25.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.729+0000 7f60b204b640 10 monclient: monmap: 2023-10-30T12:10:25.744 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:10:25.745 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:25.745 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:10:25.745 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:10:25.745 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:25.745 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:10:25.746 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:10:25.746 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:10:25.746 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:10:25.746 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:25.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.729+0000 7f60b204b640 1 librados: starting msgr at 2023-10-30T12:10:25.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.729+0000 7f60b204b640 1 librados: starting objecter 2023-10-30T12:10:25.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.729+0000 7f60b204b640 1 Processor -- start 2023-10-30T12:10:25.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 1 -- start start 2023-10-30T12:10:25.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 1 librados: setting wanted keys 2023-10-30T12:10:25.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 1 librados: calling monclient init 2023-10-30T12:10:25.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 10 monclient: init 2023-10-30T12:10:25.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:25.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:10:25.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60ac0b12d0 0x7f60ac0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:25.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 10 monclient(hunting): picked mon.noname-c con 0x7f60ac0b12d0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:10:25.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60ac0b4390 0x7f60ac13cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:25.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 10 monclient(hunting): picked mon.noname-a con 0x7f60ac0b4390 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:10:25.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60ac141910 0x7f60ac1da300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:25.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 10 monclient(hunting): picked mon.noname-b con 0x7f60ac141910 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:10:25.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:25.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac1420e0 con 0x7f60ac0b4390 2023-10-30T12:10:25.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:25.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac13d970 con 0x7f60ac141910 2023-10-30T12:10:25.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:25.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac0b66f0 con 0x7f60ac0b12d0 2023-10-30T12:10:25.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:25.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.730+0000 7f60b204b640 10 monclient(hunting): authenticate will time out at 1416.944784s 2023-10-30T12:10:25.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60ab7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60ac0b12d0 0x7f60ac0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:25.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60aaffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60ac0b4390 0x7f60ac13cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:25.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60abfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60ac141910 0x7f60ac1da300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:25.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60ab7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60ac0b12d0 0x7f60ac0b61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:48556/0 (socket says 172.21.15.3:48556) 2023-10-30T12:10:25.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60ab7fe640 1 -- 172.21.15.3:0/2475327951 learned_addr learned my addr 172.21.15.3:0/2475327951 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:10:25.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60ab7fe640 10 monclient(hunting): get_auth_request con 0x7f60ac0b12d0 auth_method 0 2023-10-30T12:10:25.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:25.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60ab7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:25.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60ab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:25.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60aaffd640 10 monclient(hunting): get_auth_request con 0x7f60ac0b4390 auth_method 0 2023-10-30T12:10:25.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:25.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60aaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:25.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60aaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:25.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60abfff640 10 monclient(hunting): get_auth_request con 0x7f60ac141910 auth_method 0 2023-10-30T12:10:25.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:25.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60abfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:25.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.731+0000 7f60abfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:25.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.732+0000 7f60abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:25.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.732+0000 7f60abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:25.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.732+0000 7f60abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:25.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.732+0000 7f60ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:25.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.732+0000 7f60ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:25.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.732+0000 7f60ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:25.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.732+0000 7f60aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:25.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.732+0000 7f60aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:25.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.732+0000 7f60aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:25.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.734+0000 7f60abfff640 10 monclient(hunting): handle_auth_done global_id 4384 payload 824 2023-10-30T12:10:25.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.734+0000 7f60abfff640 1 -- 172.21.15.3:0/2475327951 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60ac0b12d0 msgr2=0x7f60ac0b61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:25.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.734+0000 7f60abfff640 1 --2- 172.21.15.3:0/2475327951 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60ac0b12d0 0x7f60ac0b61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.734+0000 7f60abfff640 1 -- 172.21.15.3:0/2475327951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60ac0b4390 msgr2=0x7f60ac13cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:25.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.734+0000 7f60abfff640 1 --2- 172.21.15.3:0/2475327951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60ac0b4390 0x7f60ac13cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:25.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.734+0000 7f60abfff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.734+0000 7f60abfff640 1 monclient: found mon.noname-b 2023-10-30T12:10:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.734+0000 7f60abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.734+0000 7f60abfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:10:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.734+0000 7f60abfff640 1 -- 172.21.15.3:0/2475327951 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac13d510 con 0x7f60ac141910 2023-10-30T12:10:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.737+0000 7f60abfff640 1 --2- 172.21.15.3:0/2475327951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60ac141910 0x7f60ac1da300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f60a400caa0 tx=0x7f60a400cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.737+0000 7f6093fff640 10 client.?.objecter ms_handle_connect 0x7f60ac141910 2023-10-30T12:10:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.737+0000 7f6093fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f6093fff640 1 -- 172.21.15.3:0/2475327951 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a400ab40 con 0x7f60ac141910 2023-10-30T12:10:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f6093fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f6093fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:25.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f6093fff640 10 monclient: dump: 2023-10-30T12:10:25.761 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:25.761 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:25.761 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.761 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.762 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:25.762 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:25.762 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:25.762 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:25.762 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:25.762 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:25.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f6093fff640 10 monclient: _finish_auth 0 2023-10-30T12:10:25.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f6093fff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:25.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f6093fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:25.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f6093fff640 1 -- 172.21.15.3:0/2475327951 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a400e040 con 0x7f60ac141910 2023-10-30T12:10:25.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f6093fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:25.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f6093fff640 1 -- 172.21.15.3:0/2475327951 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4012400 con 0x7f60ac141910 2023-10-30T12:10:25.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f60b204b640 5 monclient: authenticate success, global_id 4384 2023-10-30T12:10:25.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f6093fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:25.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f6093fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:25.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f6093fff640 10 monclient: dump: 2023-10-30T12:10:25.765 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:10:25.765 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:10:25.765 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.765 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:10:25.765 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:25.766 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:10:25.766 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:10:25.766 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:10:25.766 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:10:25.766 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:10:25.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f60b204b640 10 monclient: _renew_subs 2023-10-30T12:10:25.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f60b204b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:25.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f60b204b640 1 -- 172.21.15.3:0/2475327951 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ac1da840 con 0x7f60ac141910 2023-10-30T12:10:25.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f60a9ffb640 4 set_mon_vals no callback set 2023-10-30T12:10:25.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f60b204b640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:25.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f60b204b640 10 monclient: _renew_subs 2023-10-30T12:10:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f60b204b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f60b204b640 1 -- 172.21.15.3:0/2475327951 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ac1dace0 con 0x7f60ac141910 2023-10-30T12:10:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.743+0000 7f60b204b640 1 librados: init done 2023-10-30T12:10:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.744+0000 7f60b204b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.744+0000 7f60b204b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.744+0000 7f60b204b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.744+0000 7f60b204b640 1 -- 172.21.15.3:0/2475327951 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60ac1daf40 con 0x7f60ac141910 2023-10-30T12:10:25.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.987+0000 7f6093fff640 1 -- 172.21.15.3:0/2475327951 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f60a4012570 con 0x7f60ac141910 2023-10-30T12:10:25.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.988+0000 7f6093fff640 1 --2- 172.21.15.3:0/2475327951 >> v2:172.21.15.92:6800/1750155735 conn(0x7f609806c810 0x7f609806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:25.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.988+0000 7f60ab7fe640 1 --2- 172.21.15.3:0/2475327951 >> v2:172.21.15.92:6800/1750155735 conn(0x7f609806c810 0x7f609806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:25.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.988+0000 7f60ab7fe640 10 monclient: get_auth_request con 0x7f609806c810 auth_method 0 2023-10-30T12:10:25.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.989+0000 7f6093fff640 10 client.4384.objecter ms_handle_connect 0x7f609806c810 2023-10-30T12:10:25.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.989+0000 7f60ab7fe640 1 --2- 172.21.15.3:0/2475327951 >> v2:172.21.15.92:6800/1750155735 conn(0x7f609806c810 0x7f609806ecd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f609c009c00 tx=0x7f609c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:25.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.991+0000 7f6093fff640 1 -- 172.21.15.3:0/2475327951 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f60a40583f0 con 0x7f60ac141910 2023-10-30T12:10:25.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.991+0000 7f6093fff640 10 client.4384.objecter ms_dispatch 0x7f60ac000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T12:10:25.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.991+0000 7f6093fff640 3 client.4384.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:10:25.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.991+0000 7f6093fff640 3 client.4384.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:10:25.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:25.991+0000 7f6093fff640 20 client.4384.objecter dump_active .. 0 homeless 2023-10-30T12:10:26.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.003+0000 7f6093fff640 1 -- 172.21.15.3:0/2475327951 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f60a405c490 con 0x7f60ac141910 2023-10-30T12:10:26.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.003+0000 7f6093fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:26.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.003+0000 7f6093fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.217+0000 7f60b204b640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-10-30T12:10:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.217+0000 7f60b204b640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-10-30T12:10:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.217+0000 7f60b204b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:10:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.217+0000 7f60b204b640 1 -- 172.21.15.3:0/2475327951 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f60ac1da840 con 0x7f60ac141910 2023-10-30T12:10:26.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.222+0000 7f6093fff640 1 -- 172.21.15.3:0/2475327951 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f60a4060640 con 0x7f60ac141910 2023-10-30T12:10:26.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.222+0000 7f6093fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-10-30T12:10:26.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.222+0000 7f6093fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:26.224 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:10:26.225 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-30T12:10:26.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 10 librados: watch_flush enter 2023-10-30T12:10:26.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 10 librados: watch_flush exit 2023-10-30T12:10:26.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 20 client.4384.objecter shutdown clearing up homeless session... 2023-10-30T12:10:26.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 10 client.4384.objecter successfully canceled tick 2023-10-30T12:10:26.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 1 -- 172.21.15.3:0/2475327951 >> v2:172.21.15.92:6800/1750155735 conn(0x7f609806c810 msgr2=0x7f609806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:26.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 1 --2- 172.21.15.3:0/2475327951 >> v2:172.21.15.92:6800/1750155735 conn(0x7f609806c810 0x7f609806ecd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f609c009c00 tx=0x7f609c009190 comp rx=0 tx=0).stop 2023-10-30T12:10:26.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 10 monclient: shutdown 2023-10-30T12:10:26.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 1 -- 172.21.15.3:0/2475327951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60ac141910 msgr2=0x7f60ac1da300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 1 --2- 172.21.15.3:0/2475327951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60ac141910 0x7f60ac1da300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f60a400caa0 tx=0x7f60a400cf60 comp rx=0 tx=0).stop 2023-10-30T12:10:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 1 -- 172.21.15.3:0/2475327951 shutdown_connections 2023-10-30T12:10:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 1 --2- 172.21.15.3:0/2475327951 >> v2:172.21.15.92:6800/1750155735 conn(0x7f609806c810 0x7f609806ecd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 1 --2- 172.21.15.3:0/2475327951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60ac141910 0x7f60ac1da300 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:26.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 1 --2- 172.21.15.3:0/2475327951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60ac0b4390 0x7f60ac13cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:26.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 1 --2- 172.21.15.3:0/2475327951 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60ac0b12d0 0x7f60ac0b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:26.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 1 -- 172.21.15.3:0/2475327951 >> 172.21.15.3:0/2475327951 conn(0x7f60ac065ed0 msgr2=0x7f60ac1c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:26.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 1 -- 172.21.15.3:0/2475327951 shutdown_connections 2023-10-30T12:10:26.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 1 -- 172.21.15.3:0/2475327951 wait complete. 2023-10-30T12:10:26.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:10:26.224+0000 7f60b204b640 1 librados: shutdown 2023-10-30T12:10:26.236 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-30T12:10:26.237 INFO:teuthology.run_tasks:Running task workunit... 2023-10-30T12:10:26.250 INFO:tasks.workunit:Pulling workunits from ref d0db9f2f95b9d6344950185071cace2a50d296eb 2023-10-30T12:10:26.251 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-30T12:10:26.251 DEBUG:teuthology.orchestra.run.smithi003:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-30T12:10:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:10:26.267 INFO:teuthology.orchestra.run.smithi003.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-30T12:10:26.267 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-30T12:10:26.322 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-30T12:10:26.323 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-30T12:10:26.377 INFO:tasks.workunit:timeout=6h 2023-10-30T12:10:26.377 INFO:tasks.workunit:cleanup=True 2023-10-30T12:10:26.378 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d0db9f2f95b9d6344950185071cace2a50d296eb 2023-10-30T12:10:26.434 INFO:tasks.workunit.client.0.smithi003.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-30T12:10:35.263 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:35.198 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:35.267 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:35.203 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:35.306 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:35.242 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:35.309 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:35.218 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:35.315 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:35.224 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:35.315 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:35.251 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:35.319 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:35.255 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:35.354 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:35.263 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:35.379 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:35.314 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:55.303 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:55.239 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:55.309 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:55.244 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:55.329 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:55.265 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:55.355 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:55.264 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:55.359 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:55.268 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:55.360 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:55.295 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:55.363 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:55.299 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:55.375 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:55.284 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:10:55.381 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:55.317 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:15.375 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:15.310 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:15.381 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:15.317 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:15.410 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:15.346 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:15.428 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:02:15.337 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:15.432 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:02:15.341 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:15.447 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:02:15.356 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:15.462 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:15.397 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:15.465 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:15.401 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:15.489 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:15.424 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:35.416 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:35.351 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:35.488 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:35.423 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:35.497 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:35.433 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:35.504 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:35.439 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:35.511 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:35.446 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:35.527 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:35.462 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:35.534 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:02:35.443 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:35.538 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:02:35.447 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:35.553 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:02:35.462 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:40.086 INFO:tasks.workunit.client.0.smithi003.stderr:Updating files: 95% (11468/12071) Updating files: 96% (11589/12071) Updating files: 97% (11709/12071) Updating files: 98% (11830/12071) Updating files: 99% (11951/12071) Updating files: 100% (12071/12071) Updating files: 100% (12071/12071), done. 2023-10-30T12:11:40.183 INFO:tasks.workunit.client.0.smithi003.stderr:Note: switching to 'd0db9f2f95b9d6344950185071cace2a50d296eb'. 2023-10-30T12:11:40.183 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-10-30T12:11:40.183 INFO:tasks.workunit.client.0.smithi003.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-30T12:11:40.183 INFO:tasks.workunit.client.0.smithi003.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-30T12:11:40.184 INFO:tasks.workunit.client.0.smithi003.stderr:state without impacting any branches by switching back to a branch. 2023-10-30T12:11:40.184 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-10-30T12:11:40.184 INFO:tasks.workunit.client.0.smithi003.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-30T12:11:40.184 INFO:tasks.workunit.client.0.smithi003.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-30T12:11:40.184 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-10-30T12:11:40.184 INFO:tasks.workunit.client.0.smithi003.stderr: git switch -c 2023-10-30T12:11:40.185 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-10-30T12:11:40.185 INFO:tasks.workunit.client.0.smithi003.stderr:Or undo this operation with: 2023-10-30T12:11:40.185 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-10-30T12:11:40.185 INFO:tasks.workunit.client.0.smithi003.stderr: git switch - 2023-10-30T12:11:40.185 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-10-30T12:11:40.186 INFO:tasks.workunit.client.0.smithi003.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-30T12:11:40.186 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-10-30T12:11:40.186 INFO:tasks.workunit.client.0.smithi003.stderr:HEAD is now at d0db9f2f95b Merge pull request #54082 from pdvian/wip-qa-suites-fix 2023-10-30T12:11:40.187 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-30T12:11:40.253 INFO:tasks.workunit.client.0.smithi003.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-30T12:11:40.255 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T12:11:40.256 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-30T12:11:40.896 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-30T12:11:40.959 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-30T12:11:41.000 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T12:11:41.001 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T12:11:41.002 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-30T12:11:41.043 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T12:11:41.046 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-30T12:11:41.046 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-30T12:11:41.102 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2023-10-30T12:11:41.103 INFO:tasks.workunit:Running workunit rados/test.sh... 2023-10-30T12:11:41.103 DEBUG:teuthology.orchestra.run.smithi003:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d0db9f2f95b9d6344950185071cace2a50d296eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2023-10-30T12:11:41.162 INFO:tasks.workunit.client.0.smithi003.stderr:+ parallel=1 2023-10-30T12:11:41.162 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' '' = --serial ']' 2023-10-30T12:11:41.163 INFO:tasks.workunit.client.0.smithi003.stderr:+ color= 2023-10-30T12:11:41.163 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -t 1 ']' 2023-10-30T12:11:41.163 INFO:tasks.workunit.client.0.smithi003.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2023-10-30T12:11:41.163 INFO:tasks.workunit.client.0.smithi003.stderr:+ declare -A pids 2023-10-30T12:11:41.163 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.164 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.164 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_aio 2023-10-30T12:11:41.164 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_aio' 2023-10-30T12:11:41.165 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_aio 2023-10-30T12:11:41.165 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.165 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_aio 2023-10-30T12:11:41.165 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91808 2023-10-30T12:11:41.166 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_aio on pid 91808' 2023-10-30T12:11:41.166 INFO:tasks.workunit.client.0.smithi003.stdout:test api_aio on pid 91808 2023-10-30T12:11:41.169 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=91808 2023-10-30T12:11:41.169 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.169 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.170 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2023-10-30T12:11:41.170 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_aio_pp 2023-10-30T12:11:41.170 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_aio_pp' 2023-10-30T12:11:41.170 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_aio_pp 2023-10-30T12:11:41.170 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.171 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.171 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.171 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.171 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.171 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.172 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.172 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.172 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.172 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.172 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.173 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.173 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.173 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.173 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.173 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.174 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.174 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.174 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.174 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.174 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.174 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.175 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.175 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.175 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.175 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.175 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_aio_pp 2023-10-30T12:11:41.176 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91815 2023-10-30T12:11:41.176 INFO:tasks.workunit.client.0.smithi003.stdout:test api_aio_pp on pid 91815 2023-10-30T12:11:41.179 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_aio_pp on pid 91815' 2023-10-30T12:11:41.180 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=91815 2023-10-30T12:11:41.180 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.180 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.180 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2023-10-30T12:11:41.180 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_io 2023-10-30T12:11:41.181 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_io' 2023-10-30T12:11:41.181 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_io 2023-10-30T12:11:41.181 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.181 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.181 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.182 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.182 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.182 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.182 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.182 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.182 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.183 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.183 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.183 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.183 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.183 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.184 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.184 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.184 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.184 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.184 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.184 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.185 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.185 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.185 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.185 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.185 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.186 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.186 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.186 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.186 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.186 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.186 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.187 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.187 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.187 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.187 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.187 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.188 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.188 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.188 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.188 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.188 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.189 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.189 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.189 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.189 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.189 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.189 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_io 2023-10-30T12:11:41.190 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91824 2023-10-30T12:11:41.190 INFO:tasks.workunit.client.0.smithi003.stdout:test api_io on pid 91824 2023-10-30T12:11:41.194 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_io on pid 91824' 2023-10-30T12:11:41.194 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=91824 2023-10-30T12:11:41.194 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.195 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.195 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2023-10-30T12:11:41.195 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_io_pp 2023-10-30T12:11:41.195 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_io_pp' 2023-10-30T12:11:41.195 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_io_pp 2023-10-30T12:11:41.196 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.196 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.196 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.196 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.196 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.196 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.197 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.197 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.197 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.197 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.197 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.198 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.198 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.198 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.198 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.198 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.199 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.199 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.199 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.199 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.199 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.200 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.200 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.200 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.200 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.200 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.200 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.201 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.201 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.201 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.201 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.201 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.202 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.202 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.202 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.202 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.202 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.203 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.203 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.203 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.203 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.203 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.203 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.204 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.204 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.204 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.204 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.204 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.205 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.205 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.205 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.205 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.205 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.205 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.206 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.206 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_io_pp 2023-10-30T12:11:41.206 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91836 2023-10-30T12:11:41.206 INFO:tasks.workunit.client.0.smithi003.stdout:test api_io_pp on pid 91836 2023-10-30T12:11:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_io_pp on pid 91836' 2023-10-30T12:11:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=91836 2023-10-30T12:11:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2023-10-30T12:11:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.214 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.214 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_asio 2023-10-30T12:11:41.214 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_asio' 2023-10-30T12:11:41.214 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.214 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.215 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.215 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_asio 2023-10-30T12:11:41.215 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.215 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.215 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.216 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.216 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.216 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.216 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.216 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.216 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.217 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.217 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.217 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.217 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.217 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.218 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.218 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.218 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.218 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.218 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.219 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.219 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.219 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.219 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.219 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.219 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.220 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.220 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.220 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.220 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.220 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.221 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.221 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.221 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.221 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.221 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.221 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.222 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.222 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.222 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.222 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.222 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.223 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.223 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.223 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.223 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.223 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.224 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.224 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.224 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.224 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.224 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.224 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.225 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.225 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.225 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.225 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.226 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.226 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.226 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.226 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.226 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.228 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.228 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.228 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.228 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_asio 2023-10-30T12:11:41.228 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.229 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.229 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.229 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.229 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.229 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91849 2023-10-30T12:11:41.230 INFO:tasks.workunit.client.0.smithi003.stdout:test api_asio on pid 91849 2023-10-30T12:11:41.235 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_asio on pid 91849' 2023-10-30T12:11:41.235 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=91849 2023-10-30T12:11:41.235 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.236 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.236 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2023-10-30T12:11:41.236 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.236 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_list 2023-10-30T12:11:41.236 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_list' 2023-10-30T12:11:41.237 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_list 2023-10-30T12:11:41.237 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.237 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.237 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.237 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.238 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.238 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.238 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.238 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.238 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.239 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.239 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.239 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.239 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.239 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.240 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.240 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.240 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.240 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.240 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.241 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.241 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.241 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.241 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.241 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.241 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.242 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.242 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.242 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.242 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.242 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.243 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.243 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.243 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.243 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.243 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.244 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.244 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.244 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.244 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.244 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.245 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.245 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.245 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.245 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.245 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.246 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.246 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.246 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.246 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.246 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.246 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.247 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.247 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.247 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.247 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.247 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.248 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.248 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.248 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.248 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.248 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.248 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.249 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.249 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.249 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.249 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.249 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.250 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.250 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.250 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.250 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.250 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.250 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.251 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.251 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.251 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.251 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.251 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.252 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.252 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.252 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.252 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.252 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.252 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.253 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.253 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.253 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.253 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/91808/exe 2023-10-30T12:11:41.253 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.254 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.254 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.254 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.254 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.254 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.255 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.255 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_list 2023-10-30T12:11:41.255 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.255 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91866 2023-10-30T12:11:41.256 INFO:tasks.workunit.client.0.smithi003.stdout:test api_list on pid 91866 2023-10-30T12:11:41.261 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_list on pid 91866' 2023-10-30T12:11:41.262 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=91866 2023-10-30T12:11:41.262 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.262 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.262 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2023-10-30T12:11:41.263 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.263 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.263 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.263 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.263 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.264 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_lock 2023-10-30T12:11:41.264 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.264 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_lock' 2023-10-30T12:11:41.264 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.264 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_lock 2023-10-30T12:11:41.265 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.265 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.265 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.265 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.265 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.265 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.266 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.266 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.266 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.266 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.266 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.267 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.267 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.267 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.267 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.267 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.267 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.268 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.268 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.268 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.268 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.268 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_aio.log 2023-10-30T12:11:41.269 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.269 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.269 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.269 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.269 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.269 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.270 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.270 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.270 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_aio: /' 2023-10-30T12:11:41.270 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.270 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.271 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.271 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.271 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.271 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.271 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_aio 2023-10-30T12:11:41.271 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.272 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.272 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.272 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.272 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.272 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.273 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.273 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.273 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.273 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.273 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.273 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.274 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.274 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.274 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.274 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.274 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.275 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.275 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.275 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.275 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.275 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.276 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.276 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.276 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.276 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.276 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.276 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.277 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.277 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.277 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.277 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.277 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.278 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.278 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.278 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.278 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.278 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.279 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.279 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.279 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.279 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.279 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.279 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.280 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.280 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.280 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.280 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.280 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.281 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.281 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.281 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.281 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.281 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.281 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.282 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.282 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.282 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.282 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.282 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.283 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.283 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.283 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.283 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.283 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.283 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.284 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.284 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/91815/exe 2023-10-30T12:11:41.284 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.284 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_lock 2023-10-30T12:11:41.284 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91886 2023-10-30T12:11:41.285 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_lock on pid 91886' 2023-10-30T12:11:41.285 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2023-10-30T12:11:41.285 INFO:tasks.workunit.client.0.smithi003.stdout:test api_lock on pid 91886 2023-10-30T12:11:41.292 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=91886 2023-10-30T12:11:41.293 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.293 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.293 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.293 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_lock_pp 2023-10-30T12:11:41.293 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.294 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.294 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_lock_pp' 2023-10-30T12:11:41.294 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.294 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.294 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.295 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.295 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.295 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.295 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.295 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.295 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.296 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.296 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.296 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.296 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.296 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.297 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.297 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.297 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.297 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.297 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.298 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.298 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.298 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.298 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.298 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.299 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.299 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.299 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.299 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.299 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.300 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.300 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.300 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.300 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.300 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.300 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.301 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.301 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.301 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.301 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.301 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.302 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.302 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.302 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.302 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.302 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.303 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.303 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.303 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.303 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.303 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.303 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.304 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.304 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.304 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.304 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.304 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.305 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.305 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.305 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.305 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.305 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.305 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.306 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.306 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.306 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.306 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.306 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.307 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.307 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_aio_pp.log 2023-10-30T12:11:41.307 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.307 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.307 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.307 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.308 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.308 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.308 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.308 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.308 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.309 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.309 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.309 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.309 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.309 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.309 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.310 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.310 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.310 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.310 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_lock_pp 2023-10-30T12:11:41.310 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.311 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.311 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.311 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.311 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_aio_pp: /' 2023-10-30T12:11:41.311 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.312 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.312 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.312 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.312 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.312 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.312 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_aio_pp 2023-10-30T12:11:41.313 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.313 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.313 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.313 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.313 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.313 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.314 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.314 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.314 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.314 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_lock_pp 2023-10-30T12:11:41.314 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.315 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.315 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.315 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91905 2023-10-30T12:11:41.315 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_lock_pp on pid 91905' 2023-10-30T12:11:41.316 INFO:tasks.workunit.client.0.smithi003.stdout:test api_lock_pp on pid 91905 2023-10-30T12:11:41.322 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=91905 2023-10-30T12:11:41.323 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2023-10-30T12:11:41.323 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.323 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.323 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.323 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.324 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.324 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.324 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.324 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_misc 2023-10-30T12:11:41.324 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_misc' 2023-10-30T12:11:41.324 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.325 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/91824/exe 2023-10-30T12:11:41.325 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_misc 2023-10-30T12:11:41.325 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.325 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.325 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.326 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.326 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.326 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.326 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.326 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.326 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.327 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.327 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.327 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.327 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.327 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.328 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.328 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.328 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.328 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.328 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.328 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.329 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.329 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.329 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.329 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.329 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.330 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.330 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.330 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.330 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.330 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.331 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.331 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.331 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.331 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.331 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.331 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.332 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.332 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.332 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.332 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.332 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.333 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.333 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.333 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.333 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.333 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.333 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.334 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.334 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.334 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.334 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.334 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.335 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.335 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.335 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.335 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.335 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.335 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.336 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.336 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.336 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.336 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.336 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.337 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.337 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.337 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.337 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.337 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.338 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.338 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.338 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.338 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.338 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.339 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.339 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.339 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.339 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.339 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.339 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.340 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.340 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.340 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.340 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.340 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.341 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.341 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.341 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.341 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.341 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.341 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_misc 2023-10-30T12:11:41.342 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_io 2023-10-30T12:11:41.342 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/91836/exe 2023-10-30T12:11:41.342 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.342 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91926 2023-10-30T12:11:41.342 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_misc on pid 91926' 2023-10-30T12:11:41.343 INFO:tasks.workunit.client.0.smithi003.stdout:test api_misc on pid 91926 2023-10-30T12:11:41.361 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=91926 2023-10-30T12:11:41.362 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.362 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.362 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.362 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.363 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.363 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_io.log 2023-10-30T12:11:41.363 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.363 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.363 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.363 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.364 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.364 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.364 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.364 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2023-10-30T12:11:41.364 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.365 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.365 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.365 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.365 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.365 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.366 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.366 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.366 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.366 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.366 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_io: /' 2023-10-30T12:11:41.366 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.367 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_misc_pp 2023-10-30T12:11:41.367 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.367 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.367 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.367 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.368 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.368 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.368 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.368 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.368 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.369 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.369 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.369 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.369 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.369 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.369 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.370 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.370 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_misc_pp' 2023-10-30T12:11:41.370 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.370 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.370 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.371 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.371 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.371 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_io_pp.log 2023-10-30T12:11:41.371 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.371 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.372 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.372 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.372 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.372 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.372 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_io_pp: /' 2023-10-30T12:11:41.372 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.373 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.373 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.373 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.373 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.373 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.374 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.374 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.374 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.374 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.374 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.374 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.375 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.375 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.375 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.375 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.375 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_io_pp 2023-10-30T12:11:41.376 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.376 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.376 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.376 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.376 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.376 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.377 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.377 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.377 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.377 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.377 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.378 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.378 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.378 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.378 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.378 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.379 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.379 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.379 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.379 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.379 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.379 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.380 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.380 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.380 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.380 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.380 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.381 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.381 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.381 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.381 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.381 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.381 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.382 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.382 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.382 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.382 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.382 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.383 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.383 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.383 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.383 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.383 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.383 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.384 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.384 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.384 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.384 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.384 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.385 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.385 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.385 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.385 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.385 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.385 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.386 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.386 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.386 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.386 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.386 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.387 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.387 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.387 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.387 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.387 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.387 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.388 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.388 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.388 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.388 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.388 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.389 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.389 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.389 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.389 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.389 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.389 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.390 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.390 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.390 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.390 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.390 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.391 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.391 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.391 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.391 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.391 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.392 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.392 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.392 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.392 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.392 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.392 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.393 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.393 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.393 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.393 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.393 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.394 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.394 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.394 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.394 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.394 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.394 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.395 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.395 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.395 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.395 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.395 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.396 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.396 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.396 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.396 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.396 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.396 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.397 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.397 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.397 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.397 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.397 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.398 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.398 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.398 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.398 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.398 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.399 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.399 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.399 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.399 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.399 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.399 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.400 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.400 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.400 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/91849/exe 2023-10-30T12:11:41.400 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.400 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.401 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.401 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.401 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.401 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.401 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.402 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.402 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.402 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.402 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.402 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.402 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.403 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.403 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.403 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.403 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/91866/exe 2023-10-30T12:11:41.403 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.404 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.404 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.404 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.404 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.404 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.404 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.405 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.405 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.405 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.405 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.405 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.406 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.406 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.406 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.406 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.406 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.407 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_misc_pp 2023-10-30T12:11:41.407 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.407 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.407 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.407 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.407 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.408 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/91886/exe 2023-10-30T12:11:41.408 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.408 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.408 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.408 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.409 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.409 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.409 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.409 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.409 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.409 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_asio 2023-10-30T12:11:41.410 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.410 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.410 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.410 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.410 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.411 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.411 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_list 2023-10-30T12:11:41.411 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_list.log 2023-10-30T12:11:41.411 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.411 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.412 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.412 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.412 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.412 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.412 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.412 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.413 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.413 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.413 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_lock: /' 2023-10-30T12:11:41.413 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_misc_pp 2023-10-30T12:11:41.413 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91970 2023-10-30T12:11:41.413 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_misc_pp on pid 91970' 2023-10-30T12:11:41.414 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2023-10-30T12:11:41.414 INFO:tasks.workunit.client.0.smithi003.stdout:test api_misc_pp on pid 91970 2023-10-30T12:11:41.417 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=91970 2023-10-30T12:11:41.418 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.418 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.418 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_tier_pp 2023-10-30T12:11:41.418 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_tier_pp' 2023-10-30T12:11:41.419 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_tier_pp 2023-10-30T12:11:41.419 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.419 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.419 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.419 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.419 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.420 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.420 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.420 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.420 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.420 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.421 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.421 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.421 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.421 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.421 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.421 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.422 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.422 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.422 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.422 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.422 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.423 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.423 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.423 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.423 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.423 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.424 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_tier_pp 2023-10-30T12:11:41.424 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91977 2023-10-30T12:11:41.424 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_tier_pp on pid 91977' 2023-10-30T12:11:41.424 INFO:tasks.workunit.client.0.smithi003.stdout:test api_tier_pp on pid 91977 2023-10-30T12:11:41.437 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=91977 2023-10-30T12:11:41.438 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.438 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.438 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_asio: /' 2023-10-30T12:11:41.438 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.439 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.439 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.439 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.439 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.439 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.440 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.440 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.440 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.440 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.440 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.440 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.441 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.441 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.441 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.441 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.441 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.442 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.442 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.442 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.442 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.442 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.443 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.443 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.443 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.443 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.443 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.444 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.444 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.444 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.444 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.444 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.444 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.445 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.445 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.445 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.445 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.445 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.446 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.446 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.446 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.446 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.446 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.447 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.447 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.447 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.447 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_lock.log 2023-10-30T12:11:41.447 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_asio.log 2023-10-30T12:11:41.448 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.448 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_list: /' 2023-10-30T12:11:41.448 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.448 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.448 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.449 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.449 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.449 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.449 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.449 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.450 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.450 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.450 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.450 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.450 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.451 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.451 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.451 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.451 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.451 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.451 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.452 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.452 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.452 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.452 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.452 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.453 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.453 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.453 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_pool 2023-10-30T12:11:41.453 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_pool' 2023-10-30T12:11:41.453 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2023-10-30T12:11:41.453 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_lock 2023-10-30T12:11:41.454 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.454 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.454 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.454 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.454 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.455 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.455 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.455 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.455 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.455 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.455 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.456 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.456 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.456 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.456 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.456 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.458 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.458 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.458 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.458 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.458 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.459 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.459 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.459 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.459 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.459 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.460 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.460 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.460 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.460 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.460 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.460 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.461 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.461 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.461 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.461 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.461 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.462 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.462 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.462 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.462 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.462 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.463 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.463 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.463 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.463 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.463 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/91926/exe 2023-10-30T12:11:41.463 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.464 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.464 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_pool 2023-10-30T12:11:41.464 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.464 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.464 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.465 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.465 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.465 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.465 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.465 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.465 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.466 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.466 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.466 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.466 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.466 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.467 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.467 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.467 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.467 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.467 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.467 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.468 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.468 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.468 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.468 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.468 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.469 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.469 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.469 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_misc.log 2023-10-30T12:11:41.469 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_misc 2023-10-30T12:11:41.469 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.470 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_misc: /' 2023-10-30T12:11:41.470 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.470 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.470 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.470 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.470 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.471 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_pool 2023-10-30T12:11:41.471 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92028 2023-10-30T12:11:41.471 INFO:tasks.workunit.client.0.smithi003.stdout:test api_pool on pid 92028 2023-10-30T12:11:41.475 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_pool on pid 92028' 2023-10-30T12:11:41.475 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92028 2023-10-30T12:11:41.475 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.475 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.475 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2023-10-30T12:11:41.476 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.476 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_snapshots 2023-10-30T12:11:41.476 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.476 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_snapshots' 2023-10-30T12:11:41.476 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.476 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.477 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.477 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.477 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.477 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.477 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.478 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.478 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.478 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.478 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.478 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.479 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.479 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.479 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.479 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.479 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.479 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.480 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.480 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.480 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.480 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.480 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_snapshots 2023-10-30T12:11:41.481 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.481 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.481 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.481 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.481 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_snapshots 2023-10-30T12:11:41.482 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.482 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.482 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92058 2023-10-30T12:11:41.482 INFO:tasks.workunit.client.0.smithi003.stdout:test api_snapshots on pid 92058 2023-10-30T12:11:41.493 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_snapshots on pid 92058' 2023-10-30T12:11:41.493 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92058 2023-10-30T12:11:41.493 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.493 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.493 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.494 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.494 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.494 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.494 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.494 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.495 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.495 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.495 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.495 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.495 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.496 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.496 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.496 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.496 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.496 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.497 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.497 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.497 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.497 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.497 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/91905/exe 2023-10-30T12:11:41.498 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2023-10-30T12:11:41.498 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.498 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.498 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.498 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.499 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.499 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.499 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.499 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.499 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.499 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.502 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.502 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.502 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.502 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.502 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.503 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.503 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.503 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.503 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.503 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.504 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.504 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_snapshots_pp 2023-10-30T12:11:41.504 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.504 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.504 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.504 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.505 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.505 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.505 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_lock_pp 2023-10-30T12:11:41.505 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.505 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.506 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_snapshots_pp' 2023-10-30T12:11:41.506 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.506 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.506 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_lock_pp: /' 2023-10-30T12:11:41.506 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.507 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.507 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.507 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.507 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.507 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.507 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.508 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.508 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.508 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_lock_pp.log 2023-10-30T12:11:41.508 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.508 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.509 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.509 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.509 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.509 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.509 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.510 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.510 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.510 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.510 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.510 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.511 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.511 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.511 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.511 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.511 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.511 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.513 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.513 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.513 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.513 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.513 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.513 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.514 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.514 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.514 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.514 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.514 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.515 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.515 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.515 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.515 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.515 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.515 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_snapshots_pp 2023-10-30T12:11:41.516 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.516 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.516 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.516 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.516 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.516 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.517 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.517 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.517 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.517 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.517 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.518 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.518 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.518 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.518 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.518 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.519 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.519 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.519 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.519 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.519 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.519 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.520 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.520 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.520 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.520 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_snapshots_pp 2023-10-30T12:11:41.520 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92081 2023-10-30T12:11:41.521 INFO:tasks.workunit.client.0.smithi003.stdout:test api_snapshots_pp on pid 92081 2023-10-30T12:11:41.529 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_snapshots_pp on pid 92081' 2023-10-30T12:11:41.529 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92081 2023-10-30T12:11:41.530 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.530 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.530 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.530 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.531 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.531 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.531 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.531 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.531 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.531 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.532 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.532 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.532 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.532 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.532 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.533 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.533 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.533 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.533 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.533 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.534 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.534 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.534 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.534 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2023-10-30T12:11:41.534 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/91977/exe 2023-10-30T12:11:41.535 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.535 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.535 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.535 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.535 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.536 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.536 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.536 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.536 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.536 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.536 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.537 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.537 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.537 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.537 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.537 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.537 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.538 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.538 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.538 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.538 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.538 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.539 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.539 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.539 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.539 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.539 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.540 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.540 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.540 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.540 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.540 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.541 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.541 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.541 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.541 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.541 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.541 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_stat 2023-10-30T12:11:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_stat' 2023-10-30T12:11:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.544 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.544 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.544 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.544 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.544 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.545 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/91970/exe 2023-10-30T12:11:41.545 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.545 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.545 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.545 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_stat 2023-10-30T12:11:41.545 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.546 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.546 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.546 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.546 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.546 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.547 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.547 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.547 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.547 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.547 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.548 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.548 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.548 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.548 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.548 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.549 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.549 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.549 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.549 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.549 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.549 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_stat 2023-10-30T12:11:41.550 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.550 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92122 2023-10-30T12:11:41.550 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.550 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_stat on pid 92122' 2023-10-30T12:11:41.551 INFO:tasks.workunit.client.0.smithi003.stdout:test api_stat on pid 92122 2023-10-30T12:11:41.559 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92122 2023-10-30T12:11:41.559 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.560 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2023-10-30T12:11:41.560 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.560 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_tier_pp.log 2023-10-30T12:11:41.560 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_tier_pp: /' 2023-10-30T12:11:41.560 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.561 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.561 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.561 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.561 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.561 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.562 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.562 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.562 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.562 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.562 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.562 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.563 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.563 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.563 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.563 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.563 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_stat_pp 2023-10-30T12:11:41.564 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_stat_pp' 2023-10-30T12:11:41.564 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_misc_pp 2023-10-30T12:11:41.564 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_misc_pp.log 2023-10-30T12:11:41.564 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_misc_pp: /' 2023-10-30T12:11:41.564 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.564 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.565 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.565 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.565 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.565 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.565 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.566 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.566 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.566 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.566 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.566 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.567 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.567 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.567 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.567 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.567 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.568 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.568 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.568 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.568 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.568 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.568 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.569 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.569 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.569 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.569 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.569 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.570 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.570 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.570 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.570 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.570 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.570 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_stat_pp 2023-10-30T12:11:41.571 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.571 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.571 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.571 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.571 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.572 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.572 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.572 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.572 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.572 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.573 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.573 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.573 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.573 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.573 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.574 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_tier_pp 2023-10-30T12:11:41.574 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.574 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.574 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.574 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.575 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.575 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.575 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.575 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.575 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.575 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.576 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.576 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.576 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.576 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.576 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.577 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.577 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.577 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.577 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.577 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.577 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.578 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.578 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.578 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.578 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.578 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.579 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.579 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.579 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.579 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.579 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_stat_pp 2023-10-30T12:11:41.580 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92158 2023-10-30T12:11:41.580 INFO:tasks.workunit.client.0.smithi003.stdout:test api_stat_pp on pid 92158 2023-10-30T12:11:41.589 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_stat_pp on pid 92158' 2023-10-30T12:11:41.589 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92158 2023-10-30T12:11:41.589 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.590 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.590 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2023-10-30T12:11:41.590 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_watch_notify 2023-10-30T12:11:41.590 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_watch_notify' 2023-10-30T12:11:41.590 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.591 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.591 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.591 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.591 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.591 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.592 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.592 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.592 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.592 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.592 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.593 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.593 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.593 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.593 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.593 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.594 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.594 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.594 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.594 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.594 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.595 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.595 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.595 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.595 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.595 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.596 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.596 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.596 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.596 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.596 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.597 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.597 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.597 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.597 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.597 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.598 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.598 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.598 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.598 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.598 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.599 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.599 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.599 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.599 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.599 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.599 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.600 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.600 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.600 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_watch_notify 2023-10-30T12:11:41.600 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.600 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.601 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.601 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.601 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.601 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.601 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.602 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.602 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.602 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.602 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.602 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.602 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.603 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.603 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.603 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.603 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.603 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.604 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.604 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.604 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.604 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.604 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.605 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.605 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.605 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.605 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.605 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.605 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.606 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.606 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.606 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.606 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.606 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.607 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.607 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.607 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.607 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.607 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.608 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.608 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.608 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.608 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.608 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.609 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.609 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.609 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.609 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.609 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.610 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.610 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.610 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.610 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.610 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.610 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.611 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.611 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.611 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.611 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.611 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.612 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2023-10-30T12:11:41.612 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [----------] Global test environment set-up. 2023-10-30T12:11:41.612 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [----------] 12 tests from AsioRados 2023-10-30T12:11:41.612 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.296+0000 7f6839c38780 10 monclient: get_monmap_and_config 2023-10-30T12:11:41.613 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.296+0000 7f6839c38780 10 monclient: build_initial_monmap 2023-10-30T12:11:41.613 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.296+0000 7f6839c38780 10 monclient: monmap: 2023-10-30T12:11:41.613 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: epoch 0 2023-10-30T12:11:41.613 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:41.613 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: last_changed 0.000000 2023-10-30T12:11:41.613 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: created 0.000000 2023-10-30T12:11:41.614 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: min_mon_release 0 (unknown) 2023-10-30T12:11:41.614 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: election_strategy: 1 2023-10-30T12:11:41.614 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:11:41.614 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:11:41.614 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:11:41.615 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.615 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.297+0000 7f6839c38780 1 Processor -- start 2023-10-30T12:11:41.615 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.297+0000 7f6839c38780 1 -- start start 2023-10-30T12:11:41.615 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.297+0000 7f6839c38780 10 monclient: init 2023-10-30T12:11:41.615 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 10 monclient: _reopen_session rank -1 2023-10-30T12:11:41.616 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:11:41.616 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c089ff1d0 0x563c08a9dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:41.616 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 10 monclient(hunting): picked mon.noname-a con 0x563c089ff1d0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:11:41.616 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x563c08a019f0 0x563c089fbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:41.616 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 10 monclient(hunting): picked mon.noname-c con 0x563c08a019f0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:11:41.617 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c08a93110 0x563c089fc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:41.617 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 10 monclient(hunting): picked mon.noname-b con 0x563c08a93110 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:11:41.617 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:41.617 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c08935220 con 0x563c089ff1d0 2023-10-30T12:11:41.617 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:41.618 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c08909d20 con 0x563c08a93110 2023-10-30T12:11:41.618 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:41.618 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x563c08907160 con 0x563c08a019f0 2023-10-30T12:11:41.618 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 10 monclient(hunting): _renew_subs 2023-10-30T12:11:41.618 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.298+0000 7f6839c38780 10 monclient(hunting): authenticate will time out at 1492.512440s 2023-10-30T12:11:41.621 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.621 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.621 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.621 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.621 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.621 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.622 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.622 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.622 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.622 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.622 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.623 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.625 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.625 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.625 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.626 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.626 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.626 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.626 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.626 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.627 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.627 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.627 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.627 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.627 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.628 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.628 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_watch_notify 2023-10-30T12:11:41.628 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.628 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.628 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.629 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.629 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.629 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.629 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.629 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.629 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.630 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.630 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.630 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.630 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.630 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92208 2023-10-30T12:11:41.631 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.631 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_watch_notify on pid 92208' 2023-10-30T12:11:41.631 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.311+test api_watch_notify on pid 92208 2023-10-30T12:11:41.641 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92208 2023-10-30T12:11:41.641 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.642 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.642 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.642 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.642 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.642 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.643 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.643 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_watch_notify_pp 2023-10-30T12:11:41.643 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.643 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.643 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.644 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.644 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.644 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2023-10-30T12:11:41.644 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.644 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_watch_notify_pp' 2023-10-30T12:11:41.645 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.645 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.645 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.645 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.645 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.645 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.646 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.646 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.646 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.646 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.646 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.647 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.647 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.647 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.647 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.647 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.648 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.648 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.648 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.648 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.648 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.649 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.649 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_watch_notify_pp 2023-10-30T12:11:41.649 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.649 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.649 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.649 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.650 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.650 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.650 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.650 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.650 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.651 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.651 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.651 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.651 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.651 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.652 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.652 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.652 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.652 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.652 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.652 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.653 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.653 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.653 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.653 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92028/exe 2023-10-30T12:11:41.653 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.654 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.654 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.654 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.654 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.654 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.655 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92081/exe 2023-10-30T12:11:41.655 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.655 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.655 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.655 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.656 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.656 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.656 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.656 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.656 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.656 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.657 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.657 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.657 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.657 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.657 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.658 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.658 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.658 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.658 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.658 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.659 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.659 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.659 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.659 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.659 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.659 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.660 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.660 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.660 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.660 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.660 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.660 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92058/exe 2023-10-30T12:11:41.661 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.661 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.661 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.661 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.661 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.662 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.662 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.662 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.662 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.662 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.663 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.663 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.663 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.663 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.663 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.663 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.664 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.664 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.664 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.664 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.664 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.665 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.665 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.665 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.665 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_pool.log 2023-10-30T12:11:41.666 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2023-10-30T12:11:41.666 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.666 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.666 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.666 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.667 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_pool: /' 2023-10-30T12:11:41.667 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.667 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.667 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.667 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_pool 2023-10-30T12:11:41.668 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.668 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.668 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.668 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_snapshots 2023-10-30T12:11:41.668 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_watch_notify_pp 2023-10-30T12:11:41.669 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92236 2023-10-30T12:11:41.669 INFO:tasks.workunit.client.0.smithi003.stdout:test api_watch_notify_pp on pid 92236 2023-10-30T12:11:41.679 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_watch_notify_pp on pid 92236' 2023-10-30T12:11:41.679 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92236 2023-10-30T12:11:41.679 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.679 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.680 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_snapshots_pp 2023-10-30T12:11:41.680 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2023-10-30T12:11:41.680 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.680 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.680 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.681 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.681 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_snapshots_pp: /' 2023-10-30T12:11:41.681 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.681 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.681 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.681 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.682 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.682 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.682 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.682 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.682 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.683 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.683 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.683 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.683 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.683 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.684 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.684 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.684 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_snapshots.log 2023-10-30T12:11:41.684 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.684 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_snapshots: /' 2023-10-30T12:11:41.684 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.685 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.685 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.685 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.685 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.685 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.686 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.686 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.686 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.686 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.686 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.687 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.687 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_cmd 2023-10-30T12:11:41.687 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.687 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.687 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.688 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.688 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.688 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_cmd' 2023-10-30T12:11:41.688 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.688 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.688 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.689 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.689 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.689 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.689 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.689 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.690 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.690 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.690 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.690 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.690 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.691 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.691 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.691 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.691 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.691 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.691 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.692 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.692 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.692 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.692 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.692 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.693 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.693 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.693 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.693 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.693 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.693 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.694 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.694 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.694 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.694 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.694 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.695 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.695 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.695 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.695 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.695 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.695 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.696 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.696 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.696 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.696 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.696 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.696 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.697 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.697 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.697 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.697 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.698 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.698 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.698 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.698 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.698 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.699 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.699 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.699 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.699 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.699 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_cmd 2023-10-30T12:11:41.699 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.700 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.700 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92122/exe 2023-10-30T12:11:41.700 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.700 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.700 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_cmd 2023-10-30T12:11:41.701 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92258 2023-10-30T12:11:41.701 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.701 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.701 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.701 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.702 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.702 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.702 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.702 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.702 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.703 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.703 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.703 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.703 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.703 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.703 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.704 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.704 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.704 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.704 INFO:tasks.workunit.client.0.smithi003.stdout:test api_cmd on pid 92258 2023-10-30T12:11:41.717 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_cmd on pid 92258' 2023-10-30T12:11:41.717 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92258 2023-10-30T12:11:41.718 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.718 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.718 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_stat: /' 2023-10-30T12:11:41.718 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2023-10-30T12:11:41.719 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_stat.log 2023-10-30T12:11:41.719 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.719 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.719 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.719 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.720 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.720 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.720 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.720 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.720 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.720 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.721 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.721 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.721 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.721 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.721 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.722 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.722 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.722 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.722 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.722 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.723 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.723 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.723 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.723 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.723 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.723 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.724 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.724 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.724 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.724 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.724 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.725 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.725 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.725 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.725 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.725 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.725 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.726 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.726 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.726 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.726 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.727 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.727 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.727 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.727 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.727 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.727 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.728 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_cmd_pp 2023-10-30T12:11:41.728 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.728 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.728 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.728 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.729 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.729 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_cmd_pp' 2023-10-30T12:11:41.729 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.729 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.729 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.730 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.730 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.730 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.730 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.730 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.730 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.731 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.731 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.731 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.731 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.731 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.732 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.732 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.732 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.732 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.732 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_stat 2023-10-30T12:11:41.733 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.733 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.733 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.733 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.733 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.734 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.734 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.734 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.734 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.734 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.735 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.735 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.735 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.735 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.735 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.736 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.736 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.736 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.736 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.736 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.737 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.737 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.737 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.737 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.737 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.738 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.738 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.738 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.738 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.738 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.739 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.739 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.739 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.739 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.739 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.740 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.740 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.740 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.740 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.740 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.740 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.741 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.741 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.741 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.741 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.741 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.742 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.742 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.742 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.742 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.742 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.743 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.743 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.743 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.743 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.743 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.743 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.744 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.744 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.744 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.744 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.744 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.745 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.745 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.745 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.745 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.745 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.745 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.746 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.746 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.746 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.746 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_cmd_pp 2023-10-30T12:11:41.746 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.747 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.747 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.747 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.747 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.747 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.747 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_cmd_pp 2023-10-30T12:11:41.748 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92300 2023-10-30T12:11:41.748 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_cmd_pp on pid 92300' 2023-10-30T12:11:41.748 INFO:tasks.workunit.client.0.smithi003.stdout:test api_cmd_pp on pid 92300 2023-10-30T12:11:41.757 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92300 2023-10-30T12:11:41.758 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.758 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.758 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2023-10-30T12:11:41.758 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.758 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_service 2023-10-30T12:11:41.759 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.759 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.759 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.759 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.759 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.760 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.760 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.760 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.760 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_service' 2023-10-30T12:11:41.760 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92158/exe 2023-10-30T12:11:41.761 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.761 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.761 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.761 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.761 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.761 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.762 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.762 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.762 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_service 2023-10-30T12:11:41.762 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.762 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.762 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.763 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.763 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.763 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.763 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92208/exe 2023-10-30T12:11:41.763 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.763 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.764 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.764 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.764 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.764 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.764 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.764 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.765 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.765 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.765 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.765 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.765 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.766 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.766 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.766 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.766 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.766 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.767 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.767 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.767 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.767 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.767 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_stat_pp.log 2023-10-30T12:11:41.768 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.768 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.768 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_stat_pp: /' 2023-10-30T12:11:41.768 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_stat_pp 2023-10-30T12:11:41.768 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.769 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.769 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.769 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.769 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.769 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.769 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.770 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.770 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.770 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.770 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.770 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.771 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.771 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.771 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.771 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.771 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.771 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.772 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.772 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.772 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.772 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.772 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.773 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.773 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.773 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.773 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.773 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.773 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.774 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.774 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.774 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.774 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.774 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.775 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.775 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.775 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.775 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.775 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.776 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.776 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.776 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.776 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.776 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_watch_notify 2023-10-30T12:11:41.777 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.777 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.777 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_watch_notify.log 2023-10-30T12:11:41.777 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.777 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.778 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.778 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.778 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.778 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.779 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.779 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.779 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.779 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.779 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.780 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.780 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.780 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.780 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.780 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.781 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.781 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.781 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.781 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.781 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.781 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.782 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.782 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_watch_notify: /' 2023-10-30T12:11:41.782 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.782 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.782 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_service 2023-10-30T12:11:41.783 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92341 2023-10-30T12:11:41.783 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_service on pid 92341' 2023-10-30T12:11:41.783 INFO:tasks.workunit.client.0.smithi003.stdout:test api_service on pid 92341 2023-10-30T12:11:41.790 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92341 2023-10-30T12:11:41.790 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.790 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.790 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.791 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2023-10-30T12:11:41.791 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.791 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.791 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.791 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.792 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.792 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.792 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.792 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.792 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.793 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.793 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.793 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.793 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.793 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.794 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.794 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.794 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.794 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.794 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.794 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.795 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.795 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.795 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.795 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.795 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.796 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.796 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.796 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.796 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.796 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.797 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.797 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.797 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.797 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.797 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.798 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.798 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.798 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.798 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.798 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.799 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.799 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.799 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.799 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.799 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_service_pp 2023-10-30T12:11:41.800 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.800 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.800 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.800 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.800 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.800 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.801 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.801 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.801 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.801 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_service_pp' 2023-10-30T12:11:41.801 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.802 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92236/exe 2023-10-30T12:11:41.802 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.802 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.802 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.802 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.802 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.803 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.803 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.803 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.803 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.803 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_watch_notify_pp 2023-10-30T12:11:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2023-10-30T12:11:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_service_pp 2023-10-30T12:11:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_service_pp 2023-10-30T12:11:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.807 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.807 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92371 2023-10-30T12:11:41.807 INFO:tasks.workunit.client.0.smithi003.stdout:test api_service_pp on pid 92371 2023-10-30T12:11:41.820 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_service_pp on pid 92371' 2023-10-30T12:11:41.820 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.820 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92371 2023-10-30T12:11:41.821 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.821 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.821 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.821 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.821 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2023-10-30T12:11:41.822 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.822 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.822 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.822 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.822 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.823 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.823 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.823 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.823 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.823 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.823 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.824 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.824 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.824 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.824 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.824 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.825 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.825 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.825 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.825 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.825 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.826 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.826 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.826 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.826 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.826 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.826 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.827 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.827 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.827 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.827 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.827 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.828 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.828 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.828 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.828 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.828 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.829 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.829 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.829 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_c_write_operations 2023-10-30T12:11:41.829 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_c_write_operations' 2023-10-30T12:11:41.829 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.830 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.830 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.830 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.830 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.830 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2023-10-30T12:11:41.830 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.831 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.831 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.831 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.831 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.831 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.832 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.832 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.832 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.832 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.832 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.832 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.833 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.833 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.833 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.833 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.833 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.834 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.834 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.834 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.834 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.834 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.835 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.835 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92258/exe 2023-10-30T12:11:41.835 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.835 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.835 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.836 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.836 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.836 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.836 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.836 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.836 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.837 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.837 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.837 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.837 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.837 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.838 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.838 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.838 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.838 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_c_write_operations 2023-10-30T12:11:41.838 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.839 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.839 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.839 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.839 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.839 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.840 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.840 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.840 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.840 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.840 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.840 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.841 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.841 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.841 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.841 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.841 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.842 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.842 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.842 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.842 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.842 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.843 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.843 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92300/exe 2023-10-30T12:11:41.843 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.843 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.843 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.843 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.844 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.844 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.844 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.844 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.844 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.845 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.845 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.845 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.845 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.845 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.845 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.846 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.846 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.846 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.846 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.846 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.847 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.847 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.847 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.847 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.847 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.847 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.848 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.848 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.848 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.848 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.848 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.849 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_cmd: /' 2023-10-30T12:11:41.849 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.849 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.849 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.849 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.849 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.850 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.850 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_cmd 2023-10-30T12:11:41.850 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.850 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.850 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.851 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.851 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_cmd_pp 2023-10-30T12:11:41.851 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.851 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2023-10-30T12:11:41.851 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.852 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.852 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.852 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.852 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.852 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.852 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_cmd.log 2023-10-30T12:11:41.853 INFO:tasks.workunit.client.0.smithi003.stdout:test api_c_write_operations on pid 92410 2023-10-30T12:11:41.857 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_c_write_operations 2023-10-30T12:11:41.858 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92410 2023-10-30T12:11:41.858 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_c_write_operations on pid 92410' 2023-10-30T12:11:41.858 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92410 2023-10-30T12:11:41.858 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.859 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.859 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2023-10-30T12:11:41.859 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.859 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.859 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.860 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.860 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.860 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.860 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_c_read_operations 2023-10-30T12:11:41.860 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_c_read_operations' 2023-10-30T12:11:41.860 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_cmd_pp: /' 2023-10-30T12:11:41.861 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.861 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.861 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.861 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.861 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.862 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.862 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.862 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.862 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.862 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.863 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.863 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.863 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.863 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.863 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.863 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.864 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.864 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.864 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.864 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.864 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.865 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.865 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.865 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.865 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.865 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.865 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.866 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.866 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.866 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.866 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.866 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.867 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.867 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.867 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.867 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.867 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.868 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.868 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.868 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.868 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.868 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.869 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.869 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.869 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.869 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.869 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_c_read_operations 2023-10-30T12:11:41.869 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.870 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.870 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.870 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.870 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.870 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.871 INFO:tasks.workunit.client.0.smithi003.stdout:0000 7f6836f27640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c089ff1d0 0x563c08a9dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:41.871 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.325+0000 7f6836f27640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c089ff1d0 0x563c08a9dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40208/0 (socket says 172.21.15.3:40208) 2023-10-30T12:11:41.871 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.325+0000 7f6836f27640 1 -- 172.21.15.3:0/4274034004 learned_addr learned my addr 172.21.15.3:0/4274034004 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:11:41.871 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.325+0000 7f6836f27640 10 monclient(hunting): get_auth_request con 0x563c089ff1d0 auth_method 0 2023-10-30T12:11:41.872 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.325+0000 7f6836f27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:41.872 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.325+0000 7f6836f27640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:41.872 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.325+0000 7f6836f27640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:41.872 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.327+0000 7f6836f27640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:41.872 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.327+0000 7f6836f27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:41.873 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.327+0000 7f6836f27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:41.873 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.329+0000 7f6837728640 1 --2- 172.21.15.3:0/4274034004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c08a93110 0x563c089fc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:41.873 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.330+0000 7f6836f27640 10 monclient(hunting): handle_auth_done global_id 4434 payload 274 2023-10-30T12:11:41.873 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.330+0000 7f6836f27640 1 -- 172.21.15.3:0/4274034004 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x563c08a019f0 msgr2=0x563c089fbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:41.873 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.330+0000 7f6836f27640 1 --2- 172.21.15.3:0/4274034004 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x563c08a019f0 0x563c089fbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:41.874 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.330+0000 7f6836f27640 1 -- 172.21.15.3:0/4274034004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c08a93110 msgr2=0x563c089fc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:41.874 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.330+0000 7f6836f27640 1 --2- 172.21.15.3:0/4274034004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c08a93110 0x563c089fc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:41.874 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.330+0000 7f6836f27640 10 monclient: _finish_hunting 0 2023-10-30T12:11:41.874 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.330+0000 7f6836f27640 1 monclient: found mon.noname-a 2023-10-30T12:11:41.874 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.330+0000 7f6836f27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:41.875 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.330+0000 7f6836f27640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:11:41.875 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.330+0000 7f6836f27640 1 -- 172.21.15.3:0/4274034004 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c08a8f130 con 0x563c089ff1d0 2023-10-30T12:11:41.876 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.876 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_c_read_operations 2023-10-30T12:11:41.876 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.876 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.877 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.877 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.877 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.877 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.877 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92440 2023-10-30T12:11:41.878 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.438+0000 7f6836f27640 1 --2- 172.21.15.3:0/4274034004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c089ff1d0 0x5test api_c_read_operations on pid 92440 2023-10-30T12:11:41.885 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_c_read_operations on pid 92440' 2023-10-30T12:11:41.885 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92440 2023-10-30T12:11:41.885 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.886 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.886 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.886 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.886 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.886 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.887 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.887 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.887 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.887 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.887 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.888 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.888 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.888 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.888 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.888 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2023-10-30T12:11:41.888 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.889 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.889 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s api_cls_remote_reads 2023-10-30T12:11:41.889 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.889 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.889 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' api_cls_remote_reads' 2023-10-30T12:11:41.890 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.890 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.890 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.890 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.890 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.890 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.891 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.891 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.891 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.891 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.891 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.892 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.892 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.892 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.892 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.892 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.893 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.893 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.893 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.893 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.893 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.893 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.894 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.894 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.894 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.894 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.894 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.895 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.895 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.895 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.895 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.895 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.895 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.896 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.896 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.896 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.896 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.896 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.897 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.897 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.897 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.897 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.897 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.898 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.898 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.898 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.898 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.898 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.898 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.899 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.899 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.899 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.899 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.899 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.900 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.900 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.900 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.900 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.900 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.901 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.901 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.901 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.901 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.901 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.901 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.902 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.902 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.902 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.902 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.902 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.903 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.903 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.903 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.903 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.903 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.904 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.904 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.904 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92341/exe 2023-10-30T12:11:41.904 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo api_cls_remote_reads 2023-10-30T12:11:41.904 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=api_cls_remote_reads 2023-10-30T12:11:41.904 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92470 2023-10-30T12:11:41.905 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test api_cls_remote_reads on pid 92470' 2023-10-30T12:11:41.905 INFO:tasks.workunit.client.0.smithi003.stdout:test api_cls_remote_reads on pid 92470 2023-10-30T12:11:41.920 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92470 2023-10-30T12:11:41.920 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.920 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.920 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.920 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.921 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.921 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.921 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.921 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.921 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.922 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.922 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.922 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.922 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.922 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.923 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.923 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.923 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.923 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.923 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.924 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.924 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.924 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.924 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.924 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.924 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.925 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.925 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.925 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.925 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.925 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.926 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.926 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.926 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.926 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.926 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.927 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.927 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.927 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.927 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.927 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.927 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.928 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.928 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.928 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.928 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.928 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.929 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.929 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.929 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.929 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.929 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.930 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.930 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.930 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.930 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.930 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.930 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.931 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.931 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.931 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.931 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.931 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92371/exe 2023-10-30T12:11:41.932 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.932 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.932 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.932 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.932 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.932 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.933 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.933 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.933 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.933 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.933 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.934 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2023-10-30T12:11:41.934 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.934 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.934 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.934 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_service 2023-10-30T12:11:41.934 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.935 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.935 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.935 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.935 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.935 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.936 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.936 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.936 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.936 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.936 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.936 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.937 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.937 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.937 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.937 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.937 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.938 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.938 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.938 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.938 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.938 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.938 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.939 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.939 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.939 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.939 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_service: /' 2023-10-30T12:11:41.939 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.940 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.940 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s list_parallel 2023-10-30T12:11:41.940 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_service.log 2023-10-30T12:11:41.940 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' list_parallel' 2023-10-30T12:11:41.940 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:41.940 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:41.941 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:41.941 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.941 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:41.941 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.941 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.942 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:41.942 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:41.942 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.942 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:41.942 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.943 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:41.943 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.943 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:41.943 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.943 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:41.943 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:41.944 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:41.944 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_service_pp.log 2023-10-30T12:11:41.944 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_service_pp: /' 2023-10-30T12:11:41.944 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.944 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.945 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.945 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.945 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.945 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.945 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.945 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.946 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.946 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.946 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.946 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.946 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.947 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.947 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.947 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.947 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.947 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.947 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.948 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.948 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.948 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.948 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_service_pp 2023-10-30T12:11:41.948 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.949 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:41.949 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.949 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:41.949 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:41.949 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:41.950 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo list_parallel 2023-10-30T12:11:41.950 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.950 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:41.950 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:41.950 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:41.951 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:41.951 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:41.951 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.951 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:41.951 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:41.951 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:41.952 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:41.952 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:41.952 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.952 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:41.952 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.953 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.953 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.953 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:41.953 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.953 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:41.953 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.954 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:41.954 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.954 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:41.954 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.954 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:41.955 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.955 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.955 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.955 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.955 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.955 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.956 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.956 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.956 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.956 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.956 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.957 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92410/exe 2023-10-30T12:11:41.957 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.957 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.957 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.957 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92440/exe 2023-10-30T12:11:41.957 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=list_parallel 2023-10-30T12:11:41.958 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92515 2023-10-30T12:11:41.958 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.958 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test list_parallel on pid 92515' 2023-10-30T12:11:41.958 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:41.958 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2023-10-30T12:11:41.959 INFO:tasks.workunit.client.0.smithi003.stdout:test list_parallel on pid 92515 2023-10-30T12:11:41.964 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92515 2023-10-30T12:11:41.965 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.965 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.965 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.965 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.966 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.966 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.966 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.966 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.966 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.967 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.967 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.967 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.967 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.967 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.967 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.968 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.968 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_c_write_operations 2023-10-30T12:11:41.968 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.968 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2023-10-30T12:11:41.968 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.969 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s open_pools_parallel 2023-10-30T12:11:41.969 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.969 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' open_pools_parallel' 2023-10-30T12:11:41.969 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.969 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.969 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.970 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.970 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.970 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.970 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.970 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.971 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.971 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.971 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.971 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.971 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.972 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.972 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.972 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.972 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.972 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.972 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.973 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo open_pools_parallel 2023-10-30T12:11:41.973 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_c_write_operations: /' 2023-10-30T12:11:41.973 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.973 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.973 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.974 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.974 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.974 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.974 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.974 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.975 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.975 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.975 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.975 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.975 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.975 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.976 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.976 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.976 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.976 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.976 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.977 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.977 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=open_pools_parallel 2023-10-30T12:11:41.977 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92529 2023-10-30T12:11:41.977 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test open_pools_parallel on pid 92529' 2023-10-30T12:11:41.978 INFO:tasks.workunit.client.0.smithi003.stdout:test open_pools_parallel on pid 92529 2023-10-30T12:11:41.983 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92529 2023-10-30T12:11:41.984 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2023-10-30T12:11:41.984 INFO:tasks.workunit.client.0.smithi003.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:11:41.984 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:41.984 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.984 INFO:tasks.workunit.client.0.smithi003.stderr:++ printf %25s delete_pools_parallel 2023-10-30T12:11:41.985 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.985 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.985 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.985 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.985 INFO:tasks.workunit.client.0.smithi003.stderr:+ r=' delete_pools_parallel' 2023-10-30T12:11:41.986 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:41.986 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:41.986 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:41.986 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:41.986 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:41.987 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:41.987 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:41.987 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:41.987 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:41.987 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:41.988 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:41.988 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:41.988 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.988 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.988 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:41.988 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:41.989 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.989 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:41.989 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.989 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.989 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.990 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:41.990 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:41.990 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.990 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_c_read_operations: /' 2023-10-30T12:11:41.990 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:41.991 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:41.991 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:41.991 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.991 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:41.991 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:41.991 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.992 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:41.992 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:41.992 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.992 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.992 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:41.993 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.993 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:41.993 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:41.993 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.993 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:41.994 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.994 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:41.994 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.994 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:41.994 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.995 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:41.995 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.995 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:41.995 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_c_read_operations 2023-10-30T12:11:41.995 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.995 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2023-10-30T12:11:41.996 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $1}' 2023-10-30T12:11:41.996 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:41.996 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:41.996 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:41.996 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.997 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:41.997 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.997 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:41.997 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:41.997 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:41.998 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo delete_pools_parallel 2023-10-30T12:11:41.998 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:41.998 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:41.998 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:41.998 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.998 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:41.999 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:41.999 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:41.999 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:41.999 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:41.999 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:41.999 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:42.000 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.000 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:42.000 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:42.000 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:42.000 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.001 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:42.001 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.001 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:42.001 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:42.001 INFO:tasks.workunit.client.0.smithi003.stderr:+ ff=delete_pools_parallel 2023-10-30T12:11:42.001 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92572 2023-10-30T12:11:42.002 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'test delete_pools_parallel on pid 92572' 2023-10-30T12:11:42.002 INFO:tasks.workunit.client.0.smithi003.stdout:test delete_pools_parallel on pid 92572 2023-10-30T12:11:42.025 INFO:tasks.workunit.client.0.smithi003.stderr:+ pids[$f]=92572 2023-10-30T12:11:42.025 INFO:tasks.workunit.client.0.smithi003.stderr:+ ret=0 2023-10-30T12:11:42.026 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:11:42.026 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:11:42.026 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91977 2023-10-30T12:11:42.026 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 91977 2023-10-30T12:11:42.027 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:42.027 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:42.027 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:42.027 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.027 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:42.027 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.028 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:42.028 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:42.028 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.028 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:42.028 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.029 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:42.029 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:42.029 INFO:tasks.workunit.client.0.smithi003.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2023-10-30T12:11:42.029 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.029 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:42.030 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.030 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:42.030 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:42.030 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:42.030 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:42.033 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.034 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:42.034 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.034 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:42.034 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:42.035 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:42.035 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:11:42.035 INFO:tasks.workunit.client.0.smithi003.stderr:+ . /etc/bashrc 2023-10-30T12:11:42.035 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z '' ']' 2023-10-30T12:11:42.035 INFO:tasks.workunit.client.0.smithi003.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:11:42.036 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.036 INFO:tasks.workunit.client.0.smithi003.stderr:++ shopt -q login_shell 2023-10-30T12:11:42.036 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:42.036 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:42.036 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:42.037 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:42.037 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:42.037 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:42.037 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:42.038 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:42.038 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:42.038 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:42.038 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.038 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:42.039 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.039 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:42.039 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:42.039 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.039 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:42.040 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.040 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:42.040 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:42.040 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.040 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:42.041 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.041 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:42.041 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:42.041 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:42.042 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:42.042 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.042 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:42.042 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.042 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:42.043 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:42.043 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:42.043 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:42.043 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:42.043 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:42.044 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:42.044 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:42.044 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:42.044 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:42.044 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:42.045 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:42.045 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:42.045 INFO:tasks.workunit.client.0.smithi003.stderr:+++ umask 2023-10-30T12:11:42.045 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:42.046 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:42.046 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.046 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:42.046 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.046 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:42.047 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:42.047 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.047 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:42.047 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.047 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:42.048 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:42.048 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:42.048 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:42.048 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:42.048 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:42.049 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:42.049 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:11:42.049 INFO:tasks.workunit.client.0.smithi003.stderr:++ SHELL=/bin/bash 2023-10-30T12:11:42.049 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.050 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:11:42.050 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.050 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:11:42.050 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:11:42.050 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.051 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:11:42.051 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.051 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:11:42.051 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:42.051 INFO:tasks.workunit.client.0.smithi003.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:11:42.052 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.052 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:11:42.052 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.052 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:11:42.053 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:42.053 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:42.053 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92470/exe 2023-10-30T12:11:42.053 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:42.053 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:42.054 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:42.054 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.054 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:42.054 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.055 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:42.055 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:42.055 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.055 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:42.056 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.056 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:42.056 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.056 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:42.056 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.057 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:42.057 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:42.057 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:42.057 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:42.058 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:42.058 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:42.058 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:42.058 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:42.058 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:42.059 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:42.059 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:42.059 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:42.059 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:42.060 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:42.060 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:42.060 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:42.060 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:42.061 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:42.061 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_api_cls_remote_reads 2023-10-30T12:11:42.061 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:42.061 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:42.062 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:42.062 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:42.062 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:42.062 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:42.062 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.063 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:42.063 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.063 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:42.063 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:42.063 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.064 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:42.064 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.064 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:42.064 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:42.065 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:42.065 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2023-10-30T12:11:42.065 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:11:42.065 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92515/exe 2023-10-30T12:11:42.065 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:11:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:11:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:11:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:11:42.067 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:11:42.067 INFO:tasks.workunit.client.0.smithi003.stderr:+++ return 2023-10-30T12:11:42.067 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.067 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:11:42.067 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:11:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export S_COLORS=auto 2023-10-30T12:11:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:+++ S_COLORS=auto 2023-10-30T12:11:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.069 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:11:42.069 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.069 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:11:42.069 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:42.069 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2023-10-30T12:11:42.070 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:42.070 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:42.070 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:42.070 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:42.070 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:42.071 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:42.071 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:42.071 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:42.071 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:42.072 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:42.072 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:42.072 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:42.072 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:42.072 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:42.073 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:42.073 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:42.073 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:42.073 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:11:42.073 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:11:42.074 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:11:42.074 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.074 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:11:42.074 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.075 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:11:42.075 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:11:42.075 INFO:tasks.workunit.client.0.smithi003.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:11:42.075 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_list_parallel 2023-10-30T12:11:42.075 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:42.076 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:42.076 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:42.076 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:42.076 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:42.076 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:42.077 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:42.077 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:42.077 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:42.077 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:42.077 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:42.078 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.078 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:42.078 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.078 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:42.078 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_list_parallel.log 2023-10-30T12:11:42.079 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:42.079 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.079 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:42.079 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.080 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:42.080 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:42.080 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ list_parallel: /' 2023-10-30T12:11:42.080 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92529/exe 2023-10-30T12:11:42.080 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:42.081 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:11:42.081 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:11:42.081 INFO:tasks.workunit.client.0.smithi003.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:11:42.081 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.081 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:11:42.082 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.082 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:11:42.082 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:11:42.082 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.083 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:11:42.083 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.083 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:11:42.083 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.083 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:11:42.084 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.084 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:11:42.084 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup 2023-10-30T12:11:42.084 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:42.084 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:42.085 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:42.085 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:42.085 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:42.085 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:42.085 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:42.086 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/locale 2023-10-30T12:11:42.086 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:42.086 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:42.086 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:42.087 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ open_pools_parallel: /' 2023-10-30T12:11:42.087 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_open_pools_parallel 2023-10-30T12:11:42.087 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2023-10-30T12:11:42.087 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:42.087 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:11:42.088 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:42.088 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:11:42.088 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:11:42.088 INFO:tasks.workunit.client.0.smithi003.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:11:42.088 INFO:tasks.workunit.client.0.smithi003.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:11:42.089 INFO:tasks.workunit.client.0.smithi003.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:11:42.089 INFO:tasks.workunit.client.0.smithi003.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:11:42.089 INFO:tasks.workunit.client.0.smithi003.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:11:42.089 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:11:42.089 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:42.090 INFO:tasks.workunit.client.0.smithi003.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:11:42.090 INFO:tasks.workunit.client.0.smithi003.stderr:+++ unset LANG_backup config 2023-10-30T12:11:42.090 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n '' ']' 2023-10-30T12:11:42.090 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:11:42.091 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:11:42.091 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.091 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:11:42.091 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.091 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:11:42.092 INFO:tasks.workunit.client.0.smithi003.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:11:42.092 INFO:tasks.workunit.client.0.smithi003.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:11:42.092 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:11:42.092 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' '' ']' 2023-10-30T12:11:42.092 INFO:tasks.workunit.client.0.smithi003.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:11:42.093 INFO:tasks.workunit.client.0.smithi003.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:11:42.093 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ readlink /proc/92572/exe 2023-10-30T12:11:42.093 INFO:tasks.workunit.client.0.smithi003.stderr:++++ basename /usr/bin/bash 2023-10-30T12:11:42.093 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_declare='declare -f' 2023-10-30T12:11:42.093 INFO:tasks.workunit.client.0.smithi003.stderr:+++ which_opt=-f 2023-10-30T12:11:42.094 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export which_declare 2023-10-30T12:11:42.094 INFO:tasks.workunit.client.0.smithi003.stderr:+++ export -f which 2023-10-30T12:11:42.094 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset i 2023-10-30T12:11:42.094 INFO:tasks.workunit.client.0.smithi003.stderr:++ unset -f pathmunge 2023-10-30T12:11:42.095 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:11:42.095 INFO:tasks.workunit.client.0.smithi003.stderr:+ export PATH 2023-10-30T12:11:42.095 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:11:42.095 INFO:tasks.workunit.client.0.smithi003.stderr:+ unset rc 2023-10-30T12:11:42.095 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph_test_rados_delete_pools_parallel 2023-10-30T12:11:42.096 INFO:tasks.workunit.client.0.smithi003.stderr:+ sed 's/^/ delete_pools_parallel: /' 2023-10-30T12:11:42.096 INFO:tasks.workunit.client.0.smithi003.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2023-10-30T12:11:42.097 INFO:tasks.workunit.client.0.smithi003.stdout:63c08a9dd90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6828005f90 tx=0x7f68280084c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=791d1970666fb07f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:42.097 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6835724640 1 -- 172.21.15.3:0/4274034004 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682800a4e0 con 0x563c089ff1d0 2023-10-30T12:11:42.097 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6835724640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:42.098 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6835724640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:11:42.098 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6835724640 10 monclient: dump: 2023-10-30T12:11:42.098 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: epoch 1 2023-10-30T12:11:42.098 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:11:42.098 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:11:42.099 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:11:42.099 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T12:11:42.099 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: election_strategy: 3 2023-10-30T12:11:42.099 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:11:42.100 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:11:42.100 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:11:42.100 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.100 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6835724640 10 monclient: _finish_auth 0 2023-10-30T12:11:42.100 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6835724640 10 monclient: _check_auth_tickets 2023-10-30T12:11:42.101 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6835724640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:42.101 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6835724640 1 -- 172.21.15.3:0/4274034004 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682800a650 con 0x563c089ff1d0 2023-10-30T12:11:42.101 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6835724640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:42.101 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6835724640 1 -- 172.21.15.3:0/4274034004 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682800aa10 con 0x563c089ff1d0 2023-10-30T12:11:42.102 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6835724640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:42.102 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6835724640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:11:42.102 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6835724640 10 monclient: dump: 2023-10-30T12:11:42.102 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: epoch 1 2023-10-30T12:11:42.103 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:11:42.103 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:11:42.103 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:11:42.103 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T12:11:42.103 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: election_strategy: 3 2023-10-30T12:11:42.104 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:11:42.104 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:11:42.104 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:11:42.104 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.104 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6839c38780 5 monclient: authenticate success, global_id 4434 2023-10-30T12:11:42.105 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6839c38780 10 monclient: get_monmap_and_config success 2023-10-30T12:11:42.105 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.595+0000 7f6839c38780 4 set_mon_vals no callback set 2023-10-30T12:11:42.105 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.596+0000 7f6839c38780 10 monclient: shutdown 2023-10-30T12:11:42.105 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.596+0000 7f6839c38780 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:42.106 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.596+0000 7f6839c38780 1 -- 172.21.15.3:0/4274034004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c089ff1d0 msgr2=0x563c08a9dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:42.106 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.596+0000 7f6839c38780 1 --2- 172.21.15.3:0/4274034004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c089ff1d0 0x563c08a9dd90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6828005f90 tx=0x7f68280084c0 comp rx=0 tx=0).stop 2023-10-30T12:11:42.106 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.596+0000 7f6839c38780 1 -- 172.21.15.3:0/4274034004 shutdown_connections 2023-10-30T12:11:42.106 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.596+0000 7f6839c38780 1 --2- 172.21.15.3:0/4274034004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c08a93110 0x563c089fc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:42.107 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.596+0000 7f6839c38780 1 --2- 172.21.15.3:0/4274034004 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x563c08a019f0 0x563c089fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:42.107 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.596+0000 7f6839c38780 1 --2- 172.21.15.3:0/4274034004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c089ff1d0 0x563c08a9dd90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:42.107 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.596+0000 7f6839c38780 1 -- 172.21.15.3:0/4274034004 >> 172.21.15.3:0/4274034004 conn(0x563c08a05890 msgr2=0x563c08a04cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:42.107 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.596+0000 7f6839c38780 1 -- 172.21.15.3:0/4274034004 shutdown_connections 2023-10-30T12:11:42.107 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.596+0000 7f6839c38780 1 -- 172.21.15.3:0/4274034004 wait complete. 2023-10-30T12:11:42.108 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.596+0000 7f6839c38780 10 monclient: build_initial_monmap 2023-10-30T12:11:42.108 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.596+0000 7f6839c38780 10 monclient: monmap: 2023-10-30T12:11:42.108 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: epoch 0 2023-10-30T12:11:42.108 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:42.108 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: last_changed 0.000000 2023-10-30T12:11:42.109 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: created 0.000000 2023-10-30T12:11:42.109 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: min_mon_release 0 (unknown) 2023-10-30T12:11:42.109 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: election_strategy: 1 2023-10-30T12:11:42.109 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:11:42.109 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:11:42.110 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:11:42.110 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.110 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 1 librados: starting msgr at 2023-10-30T12:11:42.110 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 1 librados: starting objecter 2023-10-30T12:11:42.111 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 1 Processor -- start 2023-10-30T12:11:42.111 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 1 -- start start 2023-10-30T12:11:42.111 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 1 librados: setting wanted keys 2023-10-30T12:11:42.111 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 1 librados: calling monclient init 2023-10-30T12:11:42.111 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 10 monclient: init 2023-10-30T12:11:42.112 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 10 monclient: _reopen_session rank -1 2023-10-30T12:11:42.112 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:11:42.112 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x563c089ff1d0 0x563c08a8dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:42.112 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 10 monclient(hunting): picked mon.noname-c con 0x563c089ff1d0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:11:42.112 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c08a019f0 0x563c08a84070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:42.113 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 10 monclient(hunting): picked mon.noname-b con 0x563c08a019f0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:11:42.113 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c08a93110 0x563c08a845b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:42.113 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 10 monclient(hunting): picked mon.noname-a con 0x563c08a93110 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:11:42.113 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:42.114 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.597+0000 7f6839c38780 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c08a90da0 con 0x563c08a93110 2023-10-30T12:11:42.114 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.598+0000 7f6839c38780 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:42.114 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.598+0000 7f6839c38780 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c08a99da0 con 0x563c08a019f0 2023-10-30T12:11:42.114 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.598+0000 7f6839c38780 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:42.114 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.598+0000 7f6839c38780 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x563c08a04510 con 0x563c089ff1d0 2023-10-30T12:11:42.115 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.598+0000 7f6839c38780 10 monclient(hunting): _renew_subs 2023-10-30T12:11:42.115 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.598+0000 7f6839c38780 10 monclient(hunting): authenticate will time out at 1492.812077s 2023-10-30T12:11:42.115 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.642+0000 7f6836726640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c08a019f0 0x563c08a84070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:42.115 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.643+0000 7f6836726640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c08a019f0 0x563c08a84070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36390/0 (socket says 172.21.15.3:36390) 2023-10-30T12:11:42.115 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.643+0000 7f6836726640 1 -- 172.21.15.3:0/1402585138 learned_addr learned my addr 172.21.15.3:0/1402585138 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:11:42.116 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.643+0000 7f6836726640 10 monclient(hunting): get_auth_request con 0x563c08a019f0 auth_method 0 2023-10-30T12:11:42.116 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.643+0000 7f6836726640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:42.116 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.643+0000 7f6836726640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:42.116 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.643+0000 7f6836726640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:42.117 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.644+0000 7f6836726640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:42.117 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.644+0000 7f6836726640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:42.117 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.644+0000 7f6836726640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:42.117 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.649+0000 7f6836726640 10 monclient(hunting): handle_auth_done global_id 4465 payload 824 2023-10-30T12:11:42.117 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.649+0000 7f6836726640 1 -- 172.21.15.3:0/1402585138 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x563c089ff1d0 msgr2=0x563c08a8dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:42.118 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.649+0000 7f6836726640 1 --2- 172.21.15.3:0/1402585138 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x563c089ff1d0 0x563c08a8dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:42.118 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.649+0000 7f6836726640 1 -- 172.21.15.3:0/1402585138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c08a93110 msgr2=0x563c08a845b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:42.118 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.649+0000 7f6836726640 1 --2- 172.21.15.3:0/1402585138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c08a93110 0x563c08a845b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:42.118 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.649+0000 7f6836726640 10 monclient: _finish_hunting 0 2023-10-30T12:11:42.119 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.649+0000 7f6836726640 1 monclient: found mon.noname-b 2023-10-30T12:11:42.119 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.649+0000 7f6836726640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:42.119 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.649+0000 7f6836726640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:11:42.119 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.649+0000 7f6836726640 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c089c73e0 con 0x563c08a019f0 2023-10-30T12:11:42.119 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.656+0000 7f6836726640 1 --2- 172.21.15.3:0/1402585138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c08a019f0 0x563c08a84070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f682c00a520 tx=0x7f682c04ab60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:42.120 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.656+0000 7f681ffff640 10 client.?.objecter ms_handle_connect 0x563c08a019f0 2023-10-30T12:11:42.120 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.656+0000 7f681ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:42.120 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c00a250 con 0x563c08a019f0 2023-10-30T12:11:42.120 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f681ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:42.121 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f681ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:11:42.121 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f681ffff640 10 monclient: dump: 2023-10-30T12:11:42.121 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: epoch 1 2023-10-30T12:11:42.121 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:11:42.121 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:11:42.122 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:11:42.122 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T12:11:42.122 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: election_strategy: 3 2023-10-30T12:11:42.122 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:11:42.123 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:11:42.123 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:11:42.123 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.123 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f681ffff640 10 monclient: _finish_auth 0 2023-10-30T12:11:42.123 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f681ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:11:42.124 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f681ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:42.124 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682c052440 con 0x563c08a019f0 2023-10-30T12:11:42.124 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f6839c38780 5 monclient: authenticate success, global_id 4465 2023-10-30T12:11:42.124 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f6839c38780 10 monclient: _renew_subs 2023-10-30T12:11:42.124 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f6839c38780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:11:42.125 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563c08a99a90 con 0x563c08a019f0 2023-10-30T12:11:42.125 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f681ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:42.125 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c052800 con 0x563c08a019f0 2023-10-30T12:11:42.125 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f681ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:42.126 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f681ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:11:42.126 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f681ffff640 10 monclient: dump: 2023-10-30T12:11:42.126 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: epoch 1 2023-10-30T12:11:42.126 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:11:42.126 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:11:42.127 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:11:42.127 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T12:11:42.127 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: election_strategy: 3 2023-10-30T12:11:42.127 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:11:42.128 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:11:42.128 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:11:42.128 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.128 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f6839c38780 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:42.128 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f6839c38780 10 monclient: _renew_subs 2023-10-30T12:11:42.129 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f6835724640 4 set_mon_vals no callback set 2023-10-30T12:11:42.129 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f6839c38780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:11:42.129 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c08b32d20 con 0x563c08a019f0 2023-10-30T12:11:42.129 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f6839c38780 1 librados: init done 2023-10-30T12:11:42.130 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:41.690+0000 7f6839c38780 10 librados: wait_for_osdmap waiting 2023-10-30T12:11:42.130 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.043+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f682c052970 con 0x563c08a019f0 2023-10-30T12:11:42.130 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.044+0000 7f681ffff640 1 --2- 172.21.15.3:0/1402585138 >> v2:172.21.15.92:6800/1750155735 conn(0x7f682406aa00 0x7f682406cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:42.130 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.044+0000 7f6836f27640 1 --2- 172.21.15.3:0/1402585138 >> v2:172.21.15.92:6800/1750155735 conn(0x7f682406aa00 0x7f682406cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:42.131 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.044+0000 7f6836f27640 10 monclient: get_auth_request con 0x7f682406aa00 auth_method 0 2023-10-30T12:11:42.131 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.045+0000 7f6836f27640 1 --2- 172.21.15.3:0/1402585138 >> v2:172.21.15.92:6800/1750155735 conn(0x7f682406aa00 0x7f682406cec0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6828005330 tx=0x7f6828010000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:42.131 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.045+0000 7f681ffff640 10 client.4465.objecter ms_handle_connect 0x7f682406aa00 2023-10-30T12:11:42.131 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.057+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4090+0+0 (secure 0 0 0) 0x7f682c0987c0 con 0x563c08a019f0 2023-10-30T12:11:42.131 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.057+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(22..22 src has 1..22) v4 2023-10-30T12:11:42.132 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.057+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T12:11:44.144 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.057+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding full epoch 22 2023-10-30T12:11:44.145 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.057+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:11:44.145 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.057+0000 7f6839c38780 10 librados: wait_for_osdmap done waiting 2023-10-30T12:11:44.145 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.057+0000 7f6839c38780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:11:44.145 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.057+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x563c08974e50 con 0x563c08a019f0 2023-10-30T12:11:44.145 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.256+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f682c094400 con 0x563c08a019f0 2023-10-30T12:11:44.146 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.256+0000 7f681ffff640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-10-30T12:11:44.146 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.256+0000 7f6834f23640 10 client.4465.objecter _get_latest_version latest 22, have it 2023-10-30T12:11:44.146 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.256+0000 7f6839c38780 10 client.4465.objecter create_pool name=ceph_test_rados_api_asio 2023-10-30T12:11:44.146 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.256+0000 7f6839c38780 10 client.4465.objecter pool_op_submit 1 2023-10-30T12:11:44.147 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.256+0000 7f6839c38780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:11:44.147 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:42.256+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x563c089860d0 con 0x563c08a019f0 2023-10-30T12:11:44.147 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:43.937+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v24) v1 ==== 43+0+0 (secure 0 0 0) 0x7f682c057020 con 0x563c08a019f0 2023-10-30T12:11:44.147 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:43.937+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 pool_op_reply(tid 1 (0) Success v24) v1 2023-10-30T12:11:44.147 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:43.937+0000 7f681ffff640 10 client.4465.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v24) v1 2023-10-30T12:11:44.148 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:43.937+0000 7f681ffff640 10 client.4465.objecter have request 1 at 0x563c08a99c60 Op: create 2023-10-30T12:11:44.148 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:43.937+0000 7f681ffff640 20 client.4465.objecter waiting for client to reach epoch 24 before calling back 2023-10-30T12:11:44.148 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:43.937+0000 7f681ffff640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:44.148 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:43.937+0000 7f681ffff640 10 monclient: _renew_subs 2023-10-30T12:11:44.148 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:43.937+0000 7f681ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:11:44.148 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:43.937+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=23}) v3 -- 0x7f682c094b90 con 0x563c08a019f0 2023-10-30T12:11:44.149 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:43.937+0000 7f681ffff640 10 client.4465.objecter done 2023-10-30T12:11:44.149 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 8 ==== osd_map(23..24 src has 1..24) v4 ==== 13519+0+0 (secure 0 0 0) 0x7f682c0980c0 con 0x563c08a019f0 2023-10-30T12:11:44.149 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(23..24 src has 1..24) v4 2023-10-30T12:11:44.149 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [23,24] > 22 2023-10-30T12:11:44.149 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 23 2023-10-30T12:11:45.599 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 24 2023-10-30T12:11:45.600 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:11:45.600 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f6839c38780 10 librados: set snap read head -> 1 2023-10-30T12:11:45.600 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f6839c38780 10 librados: writefull oid=exist nspace= 2023-10-30T12:11:45.600 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08a84af0 2023-10-30T12:11:45.600 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 24 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:11:45.601 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f6839c38780 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:11:45.601 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.d3a59201 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:11:45.601 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f6839c38780 1 --2- 172.21.15.3:0/1402585138 >> v2:172.21.15.3:6804/1445650280 conn(0x563c089c5aa0 0x563c08b461e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:45.601 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:11:45.601 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f6839c38780 10 client.4465.objecter _op_submit oid exist '@24' '@24' [writefull 0~5 in=5b] tid 2 osd.1 2023-10-30T12:11:45.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:11:45.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 2 2023-10-30T12:11:45.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f6839c38780 15 client.4465.objecter _send_op 2 to 24.1 on osd.1 2023-10-30T12:11:45.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:2 24.1 24:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) v8 -- 0x563c08b46720 con 0x563c089c5aa0 2023-10-30T12:11:45.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:11:45.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.142+0000 7f6839c38780 5 client.4465.objecter 1 in flight 2023-10-30T12:11:45.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.338+0000 7f6837728640 1 --2- 172.21.15.3:0/1402585138 >> v2:172.21.15.3:6804/1445650280 conn(0x563c089c5aa0 0x563c08b461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:45.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.369+0000 7f6837728640 10 monclient: get_auth_request con 0x563c089c5aa0 auth_method 0 2023-10-30T12:11:45.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.598+0000 7f681effd640 10 monclient: tick 2023-10-30T12:11:45.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.598+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:45.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.598+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:45.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.598+0000 7f681effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:45.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.796+0000 7f681ffff640 10 client.4465.objecter ms_handle_connect 0x563c089c5aa0 2023-10-30T12:11:45.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:44.796+0000 7f6837728640 1 --2- 172.21.15.3:0/1402585138 >> v2:172.21.15.3:6804/1445650280 conn(0x563c089c5aa0 0x563c08b461e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:45.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:45.598+0000 7f681effd640 10 monclient: tick 2023-10-30T12:11:45.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:45.598+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:52.137 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:45.598+0000 7f681effd640 2 api_service: [==========] Running 4 tests from 1 test suite. 2023-10-30T12:11:52.138 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: [----------] Global test environment set-up. 2023-10-30T12:11:52.138 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: [----------] 4 tests from LibRadosService 2023-10-30T12:11:52.138 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2023-10-30T12:11:52.139 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5290 ms) 2023-10-30T12:11:52.139 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2023-10-30T12:11:52.139 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: [ OK ] LibRadosService.RegisterLate (152 ms) 2023-10-30T12:11:52.139 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2023-10-30T12:11:52.139 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: cluster: 2023-10-30T12:11:52.139 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: id: 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:11:52.140 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: health: HEALTH_WARN 2023-10-30T12:11:52.140 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 5 pool(s) do not have an application enabled 2023-10-30T12:11:52.140 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 2023-10-30T12:11:52.140 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: services: 2023-10-30T12:11:52.140 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-10-30T12:11:52.141 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: mgr: x(active, since 2m), standbys: y 2023-10-30T12:11:52.141 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: osd: 8 osds: 8 up (since 99s), 8 in (since 2m) 2023-10-30T12:11:52.141 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: laundry: 2 daemons active (1 hosts) 2023-10-30T12:11:52.141 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 2023-10-30T12:11:52.141 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: data: 2023-10-30T12:11:52.141 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: pools: 24 pools, 713 pgs 2023-10-30T12:11:52.142 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: objects: 2 objects, 449 KiB 2023-10-30T12:11:52.142 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: usage: 213 MiB used, 720 GiB / 720 GiB avail 2023-10-30T12:11:52.142 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: pgs: 67.181% pgs unknown 2023-10-30T12:11:52.142 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 31.557% pgs not active 2023-10-30T12:11:52.143 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 479 unknown 2023-10-30T12:11:52.143 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 160 creating+peering 2023-10-30T12:11:52.143 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 65 creating+activating 2023-10-30T12:11:52.143 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 9 active+clean 2023-10-30T12:11:52.143 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 2023-10-30T12:11:52.143 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 2023-10-30T12:11:52.144 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: cluster: 2023-10-30T12:11:52.144 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: id: 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:11:52.144 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: health: HEALTH_WARN 2023-10-30T12:11:52.144 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 10 pool(s) do not have an application enabled 2023-10-30T12:11:52.144 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 2023-10-30T12:11:52.145 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: services: 2023-10-30T12:11:52.145 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-10-30T12:11:52.145 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: mgr: x(active, since 2m), standbys: y 2023-10-30T12:11:52.145 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: osd: 8 osds: 8 up (since 101s), 8 in (since 2m) 2023-10-30T12:11:52.145 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: foo: 14 portals active (1 hosts, 3 zones) 2023-10-30T12:11:52.146 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-10-30T12:11:52.146 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 2023-10-30T12:11:52.146 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: data: 2023-10-30T12:11:52.146 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: pools: 29 pools, 849 pgs 2023-10-30T12:11:52.146 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: objects: 2 objects, 449 KiB 2023-10-30T12:11:52.147 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: usage: 213 MiB used, 720 GiB / 720 GiB avail 2023-10-30T12:11:52.147 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: pgs: 51.943% pgs unknown 2023-10-30T12:11:52.147 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 44.994% pgs not active 2023-10-30T12:11:52.147 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 441 unknown 2023-10-30T12:11:52.147 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 300 creating+peering 2023-10-30T12:11:52.148 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 82 creating+activating 2023-10-30T12:11:52.148 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 26 active+clean 2023-10-30T12:11:52.148 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 2023-10-30T12:11:52.148 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 2023-10-30T12:11:52.148 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: cluster: 2023-10-30T12:11:52.148 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: id: 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:11:52.149 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: health: HEALTH_WARN 2023-10-30T12:11:52.149 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 8 pool(s) do not have an application enabled 2023-10-30T12:11:52.149 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 2023-10-30T12:11:52.149 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: services: 2023-10-30T12:11:52.149 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-10-30T12:11:52.150 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: mgr: x(active, since 2m), standbys: y 2023-10-30T12:11:52.150 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: osd: 8 osds: 8 up (since 104s), 8 in (since 2m) 2023-10-30T12:11:52.150 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2023-10-30T12:11:52.150 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-10-30T12:11:52.150 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 2023-10-30T12:11:52.151 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: data: 2023-10-30T12:11:52.151 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: pools: 29 pools, 849 pgs 2023-10-30T12:11:55.702 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:55.637 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:55.707 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:55.643 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:55.781 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:02:55.689 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:55.783 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:55.718 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:55.787 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:55.723 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:55.790 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:02:55.699 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:55.911 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:55.847 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:56.003 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:02:55.912 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:56.189 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:56.124 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:11:56.694 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: objects: 2 objects, 449 0 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:56.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:46.598+0000 7f681effd640 10 monclient: tick 2023-10-30T12:11:56.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:46.598+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:56.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:46.598+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:56.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:46.691+0000 7f683918b640 10 client.4465.objecter tick 2023-10-30T12:11:56.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:47.598+0000 7f681effd640 10 monclient: tick 2023-10-30T12:11:56.696 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:47.598+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:56.696 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:47.598+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:56.696 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:48.598+0000 7f681effd640 10 monclient: tick 2023-10-30T12:11:56.696 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:48.598+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:56.696 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:48.598+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:56.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:49.598+0000 7f681effd640 10 monclient: tick 2023-10-30T12:11:56.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:49.598+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:56.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:49.598+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:56.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:50.598+0000 7f681effd640 10 monclient: tick 2023-10-30T12:11:56.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:50.598+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:56.698 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:50.598+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:56.698 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:51.598+0000 7f681effd640 10 monclient: tick 2023-10-30T12:11:56.698 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:51.598+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:56.698 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:51.598+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:56.698 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:51.692+0000 7f683918b640 10 client.4465.objecter tick 2023-10-30T12:11:56.699 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:52.598+0000 7f681effd640 10 monclient: tick 2023-10-30T12:11:56.699 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:52.598+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:56.699 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:52.598+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:56.699 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:53.598+0000 7f681effd640 10 monclient: tick 2023-10-30T12:11:56.699 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:53.598+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:56.699 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:53.598+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:56.700 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:54.598+0000 7f681effd640 10 monclient: tick 2023-10-30T12:11:56.700 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:54.598+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:56.700 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:54.598+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:56.700 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:54.599+0000 7f681effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:56.700 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:55.599+0000 7f681effd640 10 monclient: tick 2023-10-30T12:11:56.701 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:55.599+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:56.701 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:55.599+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:56.701 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:56.599+0000 7f681effd640 10 monclient: tick 2023-10-30T12:11:56.701 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:56.599+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:56.701 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:56.599+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:56.702 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:56.693+0000 7f683918b640 10 client.4465.objecter tick 2023-10-30T12:11:57.070 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:56.693+0000 7f683918b640 2 client.4465.objecter tid 2 on osd.1 is laggy 2023-10-30T12:11:57.070 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:56.693+0000 7f683918b640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:57.070 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:56.693+0000 7f683918b640 10 monclient: _renew_subs 2023-10-30T12:11:57.071 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:56.693+0000 7f683918b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:11:57.071 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:56.693+0000 7f683918b640 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=25}) v3 -- 0x7f6808002f90 con 0x563c08a019f0 2023-10-30T12:11:57.071 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:56.693+0000 7f683918b640 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- ping magic: 0 v1 -- 0x7f6808003330 con 0x563c089c5aa0 2023-10-30T12:11:57.071 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 9 ==== osd_map(25..33 src has 1..33) v4 ==== 12360+0+0 (secure 0 0 0) 0x7f682c09c210 con 0x563c08a019f0 2023-10-30T12:11:57.071 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(25..33 src has 1..33) v4 2023-10-30T12:11:57.072 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [25,33] > 24 2023-10-30T12:11:57.072 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 25 2023-10-30T12:11:57.072 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 10 client.4465.objecter checking op 2 2023-10-30T12:11:57.072 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 25 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:11:57.072 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:11:57.073 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 26 2023-10-30T12:11:57.073 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 10 client.4465.objecter checking op 2 2023-10-30T12:11:57.073 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 26 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:11:57.073 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:11:57.073 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 27 2023-10-30T12:11:57.074 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 10 client.4465.objecter checking op 2 2023-10-30T12:11:57.074 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 27 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:11:57.074 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:11:57.074 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 28 2023-10-30T12:11:57.074 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 10 client.4465.objecter checking op 2 2023-10-30T12:11:57.075 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 28 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:11:57.075 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:11:57.075 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 29 2023-10-30T12:11:57.075 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 10 client.4465.objecter checking op 2 2023-10-30T12:11:57.075 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 29 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:01.601 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:01.601 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 30 2023-10-30T12:12:01.601 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 10 client.4465.objecter checking op 2 2023-10-30T12:12:01.601 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 30 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:01.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:01.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 31 2023-10-30T12:12:01.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 10 client.4465.objecter checking op 2 2023-10-30T12:12:01.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 31 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:01.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:01.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 32 2023-10-30T12:12:01.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 10 client.4465.objecter checking op 2 2023-10-30T12:12:01.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 32 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:01.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:01.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 33 2023-10-30T12:12:01.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 10 client.4465.objecter checking op 2 2023-10-30T12:12:01.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 33 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:01.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:01.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:01.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.068+0000 7f681ffff640 20 client.4465.objecter 2 24.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-10-30T12:12:01.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.599+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:01.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.599+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:01.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:57.599+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:01.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:58.599+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:01.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:58.599+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:01.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:58.599+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:01.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:59.599+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:01.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:59.599+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:01.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:11:59.599+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:01.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:00.599+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:01.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:00.599+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:01.607 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:00.599+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:01.607 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.599+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:01.607 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.599+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:02.366 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.599+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:02.366 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.694+0000 7f683918b640 10 client.4465.objecter tick 2023-10-30T12:12:02.366 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.694+0000 7f683918b640 2 client.4465.objecter tid 2 on osd.1 is laggy 2023-10-30T12:12:02.366 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.694+0000 7f683918b640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:02.367 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.694+0000 7f683918b640 10 monclient: _renew_subs 2023-10-30T12:12:02.367 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.694+0000 7f683918b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:12:02.367 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.694+0000 7f683918b640 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=34}) v3 -- 0x7f6808003500 con 0x563c08a019f0 2023-10-30T12:12:02.367 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.694+0000 7f683918b640 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- ping magic: 0 v1 -- 0x7f6808003750 con 0x563c089c5aa0 2023-10-30T12:12:02.367 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.724+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 10 ==== osd_map(34..36 src has 1..36) v4 ==== 2317+0+0 (secure 0 0 0) 0x7f682c096200 con 0x563c08a019f0 2023-10-30T12:12:02.368 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.724+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(34..36 src has 1..36) v4 2023-10-30T12:12:02.368 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.725+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [34,36] > 33 2023-10-30T12:12:02.368 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.725+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 34 2023-10-30T12:12:02.368 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.725+0000 7f681ffff640 10 client.4465.objecter checking op 2 2023-10-30T12:12:02.368 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.725+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 34 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:02.369 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.725+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:02.369 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.725+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 35 2023-10-30T12:12:02.369 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.725+0000 7f681ffff640 10 client.4465.objecter checking op 2 2023-10-30T12:12:02.369 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.725+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 35 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:02.369 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.725+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:02.370 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.725+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 36 2023-10-30T12:12:02.370 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.725+0000 7f681ffff640 10 client.4465.objecter checking op 2 2023-10-30T12:12:02.370 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.725+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 36 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:02.370 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.725+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:02.370 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.725+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:02.371 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:01.725+0000 7f681ffff640 20 client.4465.objecter 2 24.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-10-30T12:12:02.371 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:02.364+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 1 ==== osd_map(25..35 src has 1..35) v4 ==== 13267+0+0 (crc 0 0 0) 0x7f6830006d20 con 0x563c089c5aa0 2023-10-30T12:12:02.371 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:02.364+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(25..35 src has 1..35) v4 2023-10-30T12:12:02.371 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:02.364+0000 7f681ffff640 3 client.4465.objecter handle_osd_map ignoring epochs [25,35] <= 36 2023-10-30T12:12:04.271 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:02.364+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:04.271 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:02.364+0000 7f681ffff640 20 client.4465.objecter 2 24.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-10-30T12:12:04.271 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:02.599+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:04.271 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:02.599+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:04.272 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:02.599+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:04.272 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:03.599+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:04.272 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:03.599+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:04.272 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:03.599+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:04.272 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 2 ==== osd_op_reply(2 exist [writefull 0~5] v35'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f68300060b0 con 0x563c089c5aa0 2023-10-30T12:12:04.272 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(2 exist [writefull 0~5] v35'1 uv1 ondisk = 0) v8 2023-10-30T12:12:04.273 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2023-10-30T12:12:04.273 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:04.273 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 2 ondisk uv 1 in 24.1 attempt 0 2023-10-30T12:12:04.273 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 0 2023-10-30T12:12:04.273 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 2 2023-10-30T12:12:04.274 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6837728640 15 client.4465.objecter _finish_op 2 2023-10-30T12:12:04.274 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.274 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 2 2023-10-30T12:12:04.274 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6837728640 5 client.4465.objecter 0 in flight 2023-10-30T12:12:04.275 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 10 librados: Objecter returned from writefull r=0 2023-10-30T12:12:04.275 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08a84af0 2023-10-30T12:12:04.275 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 36 base exist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:04.276 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:04.276 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.d3a59201 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:04.276 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.276 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 10 client.4465.objecter _op_submit oid exist '@24' '@24' [read 0~256] tid 3 osd.1 2023-10-30T12:12:04.276 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.276 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 3 2023-10-30T12:12:04.277 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 15 client.4465.objecter _send_op 3 to 24.1 on osd.1 2023-10-30T12:12:04.277 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:3 24.1 24:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x563c089c4580 con 0x563c089c5aa0 2023-10-30T12:12:04.277 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.310 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 5 client.4465.objecter 1 in flight 2023-10-30T12:12:04.310 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c089c4980 2023-10-30T12:12:04.310 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 36 base noexist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:04.310 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 20 client.4465.objecter _calc_target target noexist @24 -> pgid 24.a7bbaa1 2023-10-30T12:12:04.311 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.a7bbaa1 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:04.311 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.311 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 10 client.4465.objecter _op_submit oid noexist '@24' '@24' [read 0~256] tid 4 osd.1 2023-10-30T12:12:04.311 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.312 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 4 2023-10-30T12:12:04.312 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 15 client.4465.objecter _send_op 4 to 24.1 on osd.1 2023-10-30T12:12:04.312 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:4 24.1 24:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x563c089c4e20 con 0x563c089c5aa0 2023-10-30T12:12:04.312 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 5 2023-10-30T12:12:04.312 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.269+0000 7f6839c38780 5 client.4465.objecter 2 in flight 2023-10-30T12:12:04.313 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f68300060b0 con 0x563c089c5aa0 2023-10-30T12:12:04.313 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-10-30T12:12:04.313 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:04.313 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 3 ondisk uv 1 in 24.1 attempt 0 2023-10-30T12:12:04.313 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 5 2023-10-30T12:12:04.313 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 3 2023-10-30T12:12:04.314 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 15 client.4465.objecter _finish_op 3 2023-10-30T12:12:04.314 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 5 2023-10-30T12:12:04.314 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 3 2023-10-30T12:12:04.315 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 5 client.4465.objecter 1 in flight 2023-10-30T12:12:04.315 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f68300060b0 con 0x563c089c5aa0 2023-10-30T12:12:04.315 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:12:04.315 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:04.316 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 4 ondisk uv 0 in 24.1 attempt 0 2023-10-30T12:12:04.316 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 10 client.4465.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:12:04.316 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 0 2023-10-30T12:12:04.316 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 4 2023-10-30T12:12:04.316 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 15 client.4465.objecter _finish_op 4 2023-10-30T12:12:04.317 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.317 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 4 2023-10-30T12:12:04.317 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6837728640 5 client.4465.objecter 0 in flight 2023-10-30T12:12:04.317 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (39 ms) 2023-10-30T12:12:04.317 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2023-10-30T12:12:04.318 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c089c4980 2023-10-30T12:12:04.318 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 36 base exist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:04.318 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:04.318 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.d3a59201 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:04.318 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.319 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 10 client.4465.objecter _op_submit oid exist '@24' '@24' [read 0~256] tid 5 osd.1 2023-10-30T12:12:04.319 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.319 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 5 2023-10-30T12:12:04.319 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 15 client.4465.objecter _send_op 5 to 24.1 on osd.1 2023-10-30T12:12:04.319 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:5 24.1 24:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x563c089c51c0 con 0x563c089c5aa0 2023-10-30T12:12:04.320 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.320 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 5 client.4465.objecter 1 in flight 2023-10-30T12:12:04.320 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08a84af0 2023-10-30T12:12:04.320 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 36 base noexist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:04.320 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 20 client.4465.objecter _calc_target target noexist @24 -> pgid 24.a7bbaa1 2023-10-30T12:12:04.321 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.a7bbaa1 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:04.321 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.321 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 10 client.4465.objecter _op_submit oid noexist '@24' '@24' [read 0~256] tid 6 osd.1 2023-10-30T12:12:04.321 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.321 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 6 2023-10-30T12:12:04.321 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 15 client.4465.objecter _send_op 6 to 24.1 on osd.1 2023-10-30T12:12:04.339 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:6 24.1 24:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x563c08b46a90 con 0x563c089c5aa0 2023-10-30T12:12:04.339 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 5 2023-10-30T12:12:04.339 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.308+0000 7f6839c38780 5 client.4465.objecter 2 in flight 2023-10-30T12:12:04.340 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f68300060b0 con 0x563c089c5aa0 2023-10-30T12:12:04.340 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-10-30T12:12:04.340 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:04.340 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 5 ondisk uv 1 in 24.1 attempt 0 2023-10-30T12:12:04.340 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 5 2023-10-30T12:12:04.341 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 5 2023-10-30T12:12:04.341 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 15 client.4465.objecter _finish_op 5 2023-10-30T12:12:04.341 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 5 2023-10-30T12:12:04.341 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 5 2023-10-30T12:12:04.341 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 5 client.4465.objecter 1 in flight 2023-10-30T12:12:04.342 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f68300060b0 con 0x563c089c5aa0 2023-10-30T12:12:04.342 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:12:04.342 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:04.342 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 6 ondisk uv 0 in 24.1 attempt 0 2023-10-30T12:12:04.342 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 10 client.4465.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:12:04.343 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 0 2023-10-30T12:12:04.343 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 6 2023-10-30T12:12:04.343 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 15 client.4465.objecter _finish_op 6 2023-10-30T12:12:04.343 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.343 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 6 2023-10-30T12:12:04.344 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.337+0000 7f6837728640 5 client.4465.objecter 0 in flight 2023-10-30T12:12:04.344 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (29 ms) 2023-10-30T12:12:04.344 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2023-10-30T12:12:04.344 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08a84af0 2023-10-30T12:12:04.344 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 36 base exist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:04.344 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:04.370 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.d3a59201 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:04.371 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.371 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 10 client.4465.objecter _op_submit oid exist '@24' '@24' [read 0~256] tid 7 osd.1 2023-10-30T12:12:04.371 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.372 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 7 2023-10-30T12:12:04.372 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 15 client.4465.objecter _send_op 7 to 24.1 on osd.1 2023-10-30T12:12:04.372 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:7 24.1 24:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x563c089c4980 con 0x563c089c5aa0 2023-10-30T12:12:04.372 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.372 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 5 client.4465.objecter 1 in flight 2023-10-30T12:12:04.373 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08b87490 2023-10-30T12:12:04.373 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 36 base noexist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:04.373 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 20 client.4465.objecter _calc_target target noexist @24 -> pgid 24.a7bbaa1 2023-10-30T12:12:04.373 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.a7bbaa1 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:04.373 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.374 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 10 client.4465.objecter _op_submit oid noexist '@24' '@24' [read 0~256] tid 8 osd.1 2023-10-30T12:12:04.374 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.374 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 8 2023-10-30T12:12:04.374 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 15 client.4465.objecter _send_op 8 to 24.1 on osd.1 2023-10-30T12:12:04.374 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:8 24.1 24:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x563c08b87930 con 0x563c089c5aa0 2023-10-30T12:12:04.375 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 5 2023-10-30T12:12:04.375 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.338+0000 7f6839c38780 5 client.4465.objecter 2 in flight 2023-10-30T12:12:04.375 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f68300060b0 con 0x563c089c5aa0 2023-10-30T12:12:04.375 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-10-30T12:12:04.375 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:04.376 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 7 ondisk uv 1 in 24.1 attempt 0 2023-10-30T12:12:04.376 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 5 2023-10-30T12:12:04.376 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 7 2023-10-30T12:12:04.376 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 15 client.4465.objecter _finish_op 7 2023-10-30T12:12:04.376 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 5 2023-10-30T12:12:04.377 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 7 2023-10-30T12:12:04.377 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 5 client.4465.objecter 1 in flight 2023-10-30T12:12:04.377 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f68300060b0 con 0x563c089c5aa0 2023-10-30T12:12:04.377 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:12:04.377 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:04.378 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 8 ondisk uv 0 in 24.1 attempt 0 2023-10-30T12:12:04.378 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 10 client.4465.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:12:04.378 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 0 2023-10-30T12:12:04.378 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 8 2023-10-30T12:12:04.378 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 15 client.4465.objecter _finish_op 8 2023-10-30T12:12:04.379 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.379 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 8 2023-10-30T12:12:04.379 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6837728640 5 client.4465.objecter 0 in flight 2023-10-30T12:12:04.379 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (32 ms) 2023-10-30T12:12:04.379 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2023-10-30T12:12:04.379 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6839c38780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-10-30T12:12:04.380 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6839c38780 20 librados: queue_aio_write 0x563c08916db0 completion 0x563c089c73e0 write_seq 1 2023-10-30T12:12:04.380 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08a84af0 2023-10-30T12:12:04.380 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 36 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:04.380 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6839c38780 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:04.380 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.d3a59201 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:04.381 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.381 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6839c38780 10 client.4465.objecter _op_submit oid exist '@24' '@24' [write 0~5 in=5b] tid 9 osd.1 2023-10-30T12:12:04.381 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.381 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 9 2023-10-30T12:12:04.381 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6839c38780 15 client.4465.objecter _send_op 9 to 24.1 on osd.1 2023-10-30T12:12:04.382 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:9 24.1 24:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) v8 -- 0x563c08b47e10 con 0x563c089c5aa0 2023-10-30T12:12:04.600 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.600 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6839c38780 5 client.4465.objecter 1 in flight 2023-10-30T12:12:04.600 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.369+0000 7f6839c38780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-10-30T12:12:04.600 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 9 ==== osd_op_reply(9 exist [write 0~5] v36'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f68300060b0 con 0x563c089c5aa0 2023-10-30T12:12:04.601 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(9 exist [write 0~5] v36'2 uv2 ondisk = 0) v8 2023-10-30T12:12:04.601 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:04.601 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 9 ondisk uv 2 in 24.1 attempt 0 2023-10-30T12:12:04.601 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 0 2023-10-30T12:12:04.601 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 9 2023-10-30T12:12:04.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6837728640 15 client.4465.objecter _finish_op 9 2023-10-30T12:12:04.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 9 2023-10-30T12:12:04.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6837728640 5 client.4465.objecter 0 in flight 2023-10-30T12:12:04.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6837728640 20 librados: complete_aio_write 0x563c089c73e0 2023-10-30T12:12:04.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (229 ms) 2023-10-30T12:12:04.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2023-10-30T12:12:04.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6839c38780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-10-30T12:12:04.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6839c38780 20 librados: queue_aio_write 0x563c08916db0 completion 0x563c08b48180 write_seq 2 2023-10-30T12:12:04.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08a84af0 2023-10-30T12:12:04.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 36 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:04.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6839c38780 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:04.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.d3a59201 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:04.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6839c38780 10 client.4465.objecter _op_submit oid exist '@24' '@24' [write 0~5 in=5b] tid 10 osd.1 2023-10-30T12:12:04.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 10 2023-10-30T12:12:04.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6839c38780 15 client.4465.objecter _send_op 10 to 24.1 on osd.1 2023-10-30T12:12:04.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:10 24.1 24:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) v8 -- 0x563c08b47e10 con 0x563c089c5aa0 2023-10-30T12:12:04.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6839c38780 5 client.4465.objecter 1 in flight 2023-10-30T12:12:04.678 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.598+0000 7f6839c38780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-10-30T12:12:04.678 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.599+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:04.679 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.599+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:04.679 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.599+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:04.679 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.599+0000 7f681effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:04.679 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 10 ==== osd_op_reply(10 exist [write 0~5] v36'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f68300060b0 con 0x563c089c5aa0 2023-10-30T12:12:04.679 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(10 exist [write 0~5] v36'3 uv3 ondisk = 0) v8 2023-10-30T12:12:04.680 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:04.680 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 10 ondisk uv 3 in 24.1 attempt 0 2023-10-30T12:12:04.680 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 0 2023-10-30T12:12:04.681 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 10 2023-10-30T12:12:04.681 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6837728640 15 client.4465.objecter _finish_op 10 2023-10-30T12:12:04.681 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.681 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 10 2023-10-30T12:12:04.682 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6837728640 5 client.4465.objecter 0 in flight 2023-10-30T12:12:04.682 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6837728640 20 librados: complete_aio_write 0x563c08b48180 2023-10-30T12:12:04.682 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (78 ms) 2023-10-30T12:12:04.682 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2023-10-30T12:12:04.682 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6839c38780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-10-30T12:12:04.683 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6839c38780 20 librados: queue_aio_write 0x563c08916db0 completion 0x563c08b48030 write_seq 3 2023-10-30T12:12:04.683 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08a84af0 2023-10-30T12:12:04.683 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 36 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:04.683 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6839c38780 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:04.684 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.d3a59201 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:04.684 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.684 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6839c38780 10 client.4465.objecter _op_submit oid exist '@24' '@24' [write 0~5 in=5b] tid 11 osd.1 2023-10-30T12:12:04.684 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.684 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 11 2023-10-30T12:12:04.684 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6839c38780 15 client.4465.objecter _send_op 11 to 24.1 on osd.1 2023-10-30T12:12:04.685 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:11 24.1 24:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) v8 -- 0x563c08b6a430 con 0x563c089c5aa0 2023-10-30T12:12:04.852 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.853 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6839c38780 5 client.4465.objecter 1 in flight 2023-10-30T12:12:04.853 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.676+0000 7f6839c38780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-10-30T12:12:04.853 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 11 ==== osd_op_reply(11 exist [write 0~5] v36'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f683001c040 con 0x563c089c5aa0 2023-10-30T12:12:04.853 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(11 exist [write 0~5] v36'4 uv4 ondisk = 0) v8 2023-10-30T12:12:04.854 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:04.854 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 11 ondisk uv 4 in 24.1 attempt 0 2023-10-30T12:12:04.854 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 0 2023-10-30T12:12:04.854 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 11 2023-10-30T12:12:04.854 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6837728640 15 client.4465.objecter _finish_op 11 2023-10-30T12:12:04.855 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.855 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 11 2023-10-30T12:12:04.855 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6837728640 5 client.4465.objecter 0 in flight 2023-10-30T12:12:04.855 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6837728640 20 librados: complete_aio_write 0x563c08b48030 2023-10-30T12:12:04.855 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (175 ms) 2023-10-30T12:12:04.856 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2023-10-30T12:12:04.856 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08a84af0 2023-10-30T12:12:04.856 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 36 base exist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:04.856 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:04.856 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.d3a59201 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:04.857 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.857 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 10 client.4465.objecter _op_submit oid exist '@24' '@24' [read 0~0] tid 12 osd.1 2023-10-30T12:12:04.857 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.857 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 12 2023-10-30T12:12:04.857 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 15 client.4465.objecter _send_op 12 to 24.1 on osd.1 2023-10-30T12:12:04.858 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:12 24.1 24:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x563c08b46e00 con 0x563c089c5aa0 2023-10-30T12:12:04.858 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.858 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 5 client.4465.objecter 1 in flight 2023-10-30T12:12:04.858 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08b47280 2023-10-30T12:12:04.859 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 36 base noexist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:04.961 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 20 client.4465.objecter _calc_target target noexist @24 -> pgid 24.a7bbaa1 2023-10-30T12:12:04.961 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.a7bbaa1 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:04.962 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.962 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 10 client.4465.objecter _op_submit oid noexist '@24' '@24' [read 0~0] tid 13 osd.1 2023-10-30T12:12:04.962 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.962 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 13 2023-10-30T12:12:04.962 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 15 client.4465.objecter _send_op 13 to 24.1 on osd.1 2023-10-30T12:12:04.963 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:13 24.1 24:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x563c08b47720 con 0x563c089c5aa0 2023-10-30T12:12:04.963 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 5 2023-10-30T12:12:04.963 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.851+0000 7f6839c38780 5 client.4465.objecter 2 in flight 2023-10-30T12:12:04.963 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.923+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f683001c040 con 0x563c089c5aa0 2023-10-30T12:12:04.963 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.923+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-10-30T12:12:04.964 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.923+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:04.964 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.923+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 12 ondisk uv 4 in 24.1 attempt 0 2023-10-30T12:12:04.964 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.923+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 5 2023-10-30T12:12:04.964 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.923+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 12 2023-10-30T12:12:04.964 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.923+0000 7f6837728640 15 client.4465.objecter _finish_op 12 2023-10-30T12:12:04.965 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.923+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 5 2023-10-30T12:12:04.965 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.923+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 12 2023-10-30T12:12:04.965 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.923+0000 7f6837728640 5 client.4465.objecter 1 in flight 2023-10-30T12:12:04.965 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f683001c040 con 0x563c089c5aa0 2023-10-30T12:12:04.965 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:12:04.966 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:04.966 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 13 ondisk uv 0 in 24.1 attempt 0 2023-10-30T12:12:04.966 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6837728640 10 client.4465.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:12:04.966 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 0 2023-10-30T12:12:04.966 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 13 2023-10-30T12:12:04.967 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6837728640 15 client.4465.objecter _finish_op 13 2023-10-30T12:12:04.967 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.967 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 13 2023-10-30T12:12:04.967 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6837728640 5 client.4465.objecter 0 in flight 2023-10-30T12:12:04.967 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (108 ms) 2023-10-30T12:12:04.967 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2023-10-30T12:12:04.968 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08a84af0 2023-10-30T12:12:04.968 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 36 base exist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:04.968 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6839c38780 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:04.968 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.d3a59201 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:04.968 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.969 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6839c38780 10 client.4465.objecter _op_submit oid exist '@24' '@24' [read 0~0] tid 14 osd.1 2023-10-30T12:12:04.969 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.959+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:04.969 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 14 2023-10-30T12:12:04.969 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 15 client.4465.objecter _send_op 14 to 24.1 on osd.1 2023-10-30T12:12:04.969 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:14 24.1 24:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x563c08b46e00 con 0x563c089c5aa0 2023-10-30T12:12:04.970 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.970 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 5 client.4465.objecter 1 in flight 2023-10-30T12:12:04.970 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08b47280 2023-10-30T12:12:04.970 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 36 base noexist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:04.970 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 20 client.4465.objecter _calc_target target noexist @24 -> pgid 24.a7bbaa1 2023-10-30T12:12:04.970 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.a7bbaa1 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:04.971 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.971 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 10 client.4465.objecter _op_submit oid noexist '@24' '@24' [read 0~0] tid 15 osd.1 2023-10-30T12:12:04.971 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:04.971 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 15 2023-10-30T12:12:04.972 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 15 client.4465.objecter _send_op 15 to 24.1 on osd.1 2023-10-30T12:12:04.972 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:15 24.1 24:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x563c08b47720 con 0x563c089c5aa0 2023-10-30T12:12:04.972 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 5 2023-10-30T12:12:06.296 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:04.960+0000 7f6839c38780 5 client.4465.objecter 2 in flight 2023-10-30T12:12:06.296 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 14 ==== osd_map(37..37 src has 1..37) v4 ==== 704+0+0 (crc 0 0 0) 0x7f68300059e0 con 0x563c089c5aa0 2023-10-30T12:12:06.297 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(37..37 src has 1..37) v4 2023-10-30T12:12:06.297 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [37,37] > 36 2023-10-30T12:12:06.297 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 37 2023-10-30T12:12:06.297 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f681ffff640 10 client.4465.objecter checking op 14 2023-10-30T12:12:06.298 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 37 base exist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:06.298 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:06.298 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f681ffff640 10 client.4465.objecter checking op 15 2023-10-30T12:12:06.298 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 37 base noexist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:06.298 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 15 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f683001c040 con 0x563c089c5aa0 2023-10-30T12:12:06.299 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f681ffff640 20 client.4465.objecter _calc_target target noexist @24 -> pgid 24.a7bbaa1 2023-10-30T12:12:06.299 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:06.299 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f681ffff640 20 client.4465.objecter 14 24.d3a59201 osd.1 exist [read 0~0] 2023-10-30T12:12:06.299 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-10-30T12:12:06.300 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f681ffff640 20 client.4465.objecter 15 24.a7bbaa1 osd.1 noexist [read 0~0] 2023-10-30T12:12:06.300 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:06.300 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 14 ondisk uv 4 in 24.1 attempt 0 2023-10-30T12:12:06.300 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 5 2023-10-30T12:12:06.300 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 14 2023-10-30T12:12:06.301 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f6837728640 15 client.4465.objecter _finish_op 14 2023-10-30T12:12:06.301 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 5 2023-10-30T12:12:06.301 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 14 2023-10-30T12:12:06.301 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.411+0000 7f6837728640 5 client.4465.objecter 1 in flight 2023-10-30T12:12:06.301 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.599+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:06.302 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.599+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:06.302 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:05.599+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:06.302 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 16 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f683001c040 con 0x563c089c5aa0 2023-10-30T12:12:06.302 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:12:06.302 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:06.303 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 15 ondisk uv 0 in 24.1 attempt 0 2023-10-30T12:12:06.303 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6837728640 10 client.4465.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:12:06.303 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 0 2023-10-30T12:12:06.303 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 15 2023-10-30T12:12:06.304 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6837728640 15 client.4465.objecter _finish_op 15 2023-10-30T12:12:06.304 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:06.304 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 15 2023-10-30T12:12:06.304 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6837728640 5 client.4465.objecter 0 in flight 2023-10-30T12:12:06.305 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1335 ms) 2023-10-30T12:12:06.305 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2023-10-30T12:12:06.305 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08a84af0 2023-10-30T12:12:06.305 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 37 base exist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:06.306 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:06.306 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.d3a59201 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:06.306 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:06.306 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 10 client.4465.objecter _op_submit oid exist '@24' '@24' [read 0~0] tid 16 osd.1 2023-10-30T12:12:06.306 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:06.307 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 16 2023-10-30T12:12:06.307 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 15 client.4465.objecter _send_op 16 to 24.1 on osd.1 2023-10-30T12:12:06.307 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:16 24.1 24:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37) v8 -- 0x563c08b46e00 con 0x563c089c5aa0 2023-10-30T12:12:06.307 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:06.308 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 5 client.4465.objecter 1 in flight 2023-10-30T12:12:06.308 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08b69dc0 2023-10-30T12:12:06.308 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 37 base noexist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:06.308 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 20 client.4465.objecter _calc_target target noexist @24 -> pgid 24.a7bbaa1 2023-10-30T12:12:06.308 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.a7bbaa1 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:06.309 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:06.309 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 10 client.4465.objecter _op_submit oid noexist '@24' '@24' [read 0~0] tid 17 osd.1 2023-10-30T12:12:09.356 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:09.357 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 17 2023-10-30T12:12:09.357 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 15 client.4465.objecter _send_op 17 to 24.1 on osd.1 2023-10-30T12:12:09.357 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:17 24.1 24:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37) v8 -- 0x563c08b6a260 con 0x563c089c5aa0 2023-10-30T12:12:09.357 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 5 2023-10-30T12:12:09.358 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.294+0000 7f6839c38780 5 client.4465.objecter 2 in flight 2023-10-30T12:12:09.358 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.600+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:09.358 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.600+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:09.358 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.600+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:09.359 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:06.695+0000 7f683918b640 10 client.4465.objecter tick 2023-10-30T12:12:09.359 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:07.600+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:09.359 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:07.600+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:09.359 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:07.600+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:09.359 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:08.600+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:09.360 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:08.600+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:09.360 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:08.600+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:09.360 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 17 ==== osd_map(38..38 src has 1..38) v4 ==== 2744+0+0 (crc 0 0 0) 0x7f6830006330 con 0x563c089c5aa0 2023-10-30T12:12:09.360 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 18 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f683001c040 con 0x563c089c5aa0 2023-10-30T12:12:09.360 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(38..38 src has 1..38) v4 2023-10-30T12:12:09.361 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-10-30T12:12:09.361 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:09.361 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [38,38] > 37 2023-10-30T12:12:09.361 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 38 2023-10-30T12:12:09.361 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f681ffff640 10 client.4465.objecter checking op 16 2023-10-30T12:12:09.362 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 38 base exist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:09.362 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:09.362 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f681ffff640 10 client.4465.objecter checking op 17 2023-10-30T12:12:09.362 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 38 base noexist @24 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:12:09.362 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f681ffff640 20 client.4465.objecter _calc_target target noexist @24 -> pgid 24.a7bbaa1 2023-10-30T12:12:09.363 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:09.363 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f681ffff640 20 client.4465.objecter 16 24.d3a59201 osd.1 exist [read 0~0] 2023-10-30T12:12:09.363 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f681ffff640 20 client.4465.objecter 17 24.a7bbaa1 osd.1 noexist [read 0~0] 2023-10-30T12:12:09.363 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 16 ondisk uv 4 in 24.1 attempt 0 2023-10-30T12:12:09.363 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 5 2023-10-30T12:12:09.364 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 16 2023-10-30T12:12:09.364 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f6837728640 15 client.4465.objecter _finish_op 16 2023-10-30T12:12:09.364 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 5 2023-10-30T12:12:09.364 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 16 2023-10-30T12:12:09.365 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f6837728640 5 client.4465.objecter 1 in flight 2023-10-30T12:12:09.365 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.354+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 19 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f683001c040 con 0x563c089c5aa0 2023-10-30T12:12:09.365 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:12:09.365 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:09.366 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 17 ondisk uv 0 in 24.1 attempt 0 2023-10-30T12:12:09.366 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6837728640 10 client.4465.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:12:09.366 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 0 2023-10-30T12:12:09.366 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 17 2023-10-30T12:12:09.366 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6837728640 15 client.4465.objecter _finish_op 17 2023-10-30T12:12:09.367 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:09.367 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 17 2023-10-30T12:12:09.367 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6837728640 5 client.4465.objecter 0 in flight 2023-10-30T12:12:09.367 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (3061 ms) 2023-10-30T12:12:09.367 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2023-10-30T12:12:09.368 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6839c38780 20 librados: queue_aio_write 0x563c08916db0 completion 0x563c08b47ce0 write_seq 4 2023-10-30T12:12:09.368 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08a84af0 2023-10-30T12:12:09.368 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 38 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:09.368 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6839c38780 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:09.368 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.d3a59201 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:09.369 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:09.369 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6839c38780 10 client.4465.objecter _op_submit oid exist '@24' '@24' [writefull 0~5 in=5b] tid 18 osd.1 2023-10-30T12:12:12.237 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6839c38780 20 client.KiB 2023-10-30T12:12:12.237 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: usage: 213 MiB used, 720 GiB / 720 GiB avail 2023-10-30T12:12:12.237 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: pgs: 41.814% pgs unknown 2023-10-30T12:12:12.238 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 54.653% pgs not active 2023-10-30T12:12:12.238 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 361 creating+peering 2023-10-30T12:12:12.238 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 355 unknown 2023-10-30T12:12:12.238 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 103 creating+activating 2023-10-30T12:12:12.238 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 30 active+clean 2023-10-30T12:12:12.239 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 2023-10-30T12:12:12.239 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 2023-10-30T12:12:12.239 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5232 ms) 2023-10-30T12:12:12.239 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: [ RUN ] LibRadosService.Status 2023-10-30T12:12:12.239 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: [ OK ] LibRadosService.Status (20066 ms) 2023-10-30T12:12:12.240 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: [----------] 4 tests from LibRadosService (30740 ms total) 2023-10-30T12:12:12.240 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: 2023-10-30T12:12:12.240 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: [----------] Global test environment tear-down 2023-10-30T12:12:12.240 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: [==========] 4 tests from 1 test suite ran. (30740 ms total) 2023-10-30T12:12:12.240 INFO:tasks.workunit.client.0.smithi003.stdout: api_service: [ PASSED ] 4 tests. 2023-10-30T12:12:12.360 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2023-10-30T12:12:12.360 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: [----------] Global test environment set-up. 2023-10-30T12:12:12.361 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2023-10-30T12:12:12.361 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2023-10-30T12:12:12.361 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5278 ms) 2023-10-30T12:12:12.361 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2023-10-30T12:12:12.361 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (206 ms) 2023-10-30T12:12:12.362 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2023-10-30T12:12:12.362 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20083 ms) 2023-10-30T12:12:12.362 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2023-10-30T12:12:12.362 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: attempt 0 of 20 2023-10-30T12:12:12.362 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: failed to deregister: 2023-10-30T12:12:12.363 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: { 2023-10-30T12:12:12.363 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "epoch": 11, 2023-10-30T12:12:12.363 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "modified": "2023-10-30T12:12:07.799189+0000", 2023-10-30T12:12:12.363 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "services": { 2023-10-30T12:12:12.363 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "laundry": { 2023-10-30T12:12:12.364 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "daemons": { 2023-10-30T12:12:12.364 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "summary": "", 2023-10-30T12:12:12.364 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "close-test-pid92485": { 2023-10-30T12:12:12.364 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "start_epoch": 11, 2023-10-30T12:12:12.364 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "start_stamp": "2023-10-30T12:12:07.160908+0000", 2023-10-30T12:12:12.365 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "gid": 4689, 2023-10-30T12:12:12.365 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "addr": "172.21.15.3:0/1319147770", 2023-10-30T12:12:12.365 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "metadata": { 2023-10-30T12:12:12.365 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "arch": "x86_64", 2023-10-30T12:12:12.365 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "ceph_release": "reef", 2023-10-30T12:12:12.365 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "ceph_version": "ceph version 18.0.0-6839-gcc36dd85 (cc36dd85d142c6fdb77515cad8af108cec98a86d) reef (dev)", 2023-10-30T12:12:12.366 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "ceph_version_short": "18.0.0-6839-gcc36dd85", 2023-10-30T12:12:12.366 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2023-10-30T12:12:12.366 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "distro": "centos", 2023-10-30T12:12:12.366 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-10-30T12:12:12.366 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "distro_version": "9", 2023-10-30T12:12:12.367 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "foo": "bar", 2023-10-30T12:12:12.367 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "hostname": "smithi003", 2023-10-30T12:12:12.367 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-10-30T12:12:12.367 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "kernel_version": "5.14.0-378.el9.x86_64", 2023-10-30T12:12:12.367 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "mem_swap_kb": "0", 2023-10-30T12:12:12.368 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "mem_total_kb": "32494896", 2023-10-30T12:12:12.368 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "os": "Linux", 2023-10-30T12:12:12.368 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "this": "that" 2023-10-30T12:12:12.368 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: }, 2023-10-30T12:12:12.368 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "task_status": {} 2023-10-30T12:12:12.369 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: }, 2023-10-30T12:12:12.369 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "pid92481": { 2023-10-30T12:12:12.369 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "start_epoch": 10, 2023-10-30T12:12:12.369 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "start_stamp": "2023-10-30T12:11:52.243558+0000", 2023-10-30T12:12:12.369 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "gid": 4612, 2023-10-30T12:12:12.369 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "addr": "172.21.15.3:0/447309526", 2023-10-30T12:12:12.370 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "metadata": { 2023-10-30T12:12:12.370 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "arch": "x86_64", 2023-10-30T12:12:12.370 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "ceph_release": "reef", 2023-10-30T12:12:12.370 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "ceph_version": "ceph version 18.0.0-6839-gcc36dd85 (cc36dd85d142c6fdb77515cad8af108cec98a86d) reef (dev)", 2023-10-30T12:12:12.370 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "ceph_version_short": "18.0.0-6839-gcc36dd85", 2023-10-30T12:12:12.371 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2023-10-30T12:12:12.602 INFO:tasks.workunit.client.0.smithi003.stdout: a4465.objecter get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:12.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 18 2023-10-30T12:12:12.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6839c38780 15 client.4465.objecter _send_op 18 to 24.1 on osd.1 2023-10-30T12:12:12.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:18 24.1 24:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38) v8 -- 0x563c08b46e00 con 0x563c089c5aa0 2023-10-30T12:12:12.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:12.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.355+0000 7f6839c38780 5 client.4465.objecter 1 in flight 2023-10-30T12:12:12.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.600+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:12.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.600+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:12.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:09.600+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:12.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.364+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 20 ==== osd_map(39..40 src has 1..41) v4 ==== 3043+0+0 (crc 0 0 0) 0x7f6830005bb0 con 0x563c089c5aa0 2023-10-30T12:12:12.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.364+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(39..40 src has 1..41) v4 2023-10-30T12:12:12.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.364+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [39,40] > 38 2023-10-30T12:12:12.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.364+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 39 2023-10-30T12:12:12.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.364+0000 7f681ffff640 10 client.4465.objecter checking op 18 2023-10-30T12:12:12.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.364+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 39 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:12.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.364+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:12.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.364+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 40 2023-10-30T12:12:12.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.364+0000 7f681ffff640 10 client.4465.objecter checking op 18 2023-10-30T12:12:12.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.364+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 40 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:12.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.364+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:12.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.364+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:12.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.364+0000 7f681ffff640 20 client.4465.objecter 18 24.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-10-30T12:12:12.607 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.600+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:12.607 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.600+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:12.607 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:10.600+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:12.607 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:11.600+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:12.607 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:11.600+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:12.608 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:11.600+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:12.608 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:11.696+0000 7f683918b640 10 client.4465.objecter tick 2023-10-30T12:12:12.608 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:12.600+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:12.608 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:12.600+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:13.686 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:12.600+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:13.686 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.600+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:13.686 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.600+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:13.687 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.600+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:13.687 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 21 ==== osd_op_reply(18 exist [writefull 0~5] v40'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f683001c040 con 0x563c089c5aa0 2023-10-30T12:12:13.687 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(18 exist [writefull 0~5] v40'5 uv5 ondisk = 0) v8 2023-10-30T12:12:13.687 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:13.687 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 18 ondisk uv 5 in 24.1 attempt 0 2023-10-30T12:12:13.688 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 0 2023-10-30T12:12:13.688 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 18 2023-10-30T12:12:13.688 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6837728640 15 client.4465.objecter _finish_op 18 2023-10-30T12:12:13.688 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:13.688 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 18 2023-10-30T12:12:13.689 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6837728640 5 client.4465.objecter 0 in flight 2023-10-30T12:12:13.689 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6837728640 20 librados: complete_aio_write 0x563c08b47ce0 2023-10-30T12:12:13.689 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (4289 ms) 2023-10-30T12:12:13.689 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2023-10-30T12:12:13.689 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6839c38780 20 librados: queue_aio_write 0x563c08916db0 completion 0x563c08b47280 write_seq 5 2023-10-30T12:12:13.690 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08a84af0 2023-10-30T12:12:13.690 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 40 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:13.690 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6839c38780 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:13.690 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.d3a59201 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:13.690 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:13.691 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6839c38780 10 client.4465.objecter _op_submit oid exist '@24' '@24' [writefull 0~5 in=5b] tid 19 osd.1 2023-10-30T12:12:13.691 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:13.691 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 19 2023-10-30T12:12:13.691 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6839c38780 15 client.4465.objecter _send_op 19 to 24.1 on osd.1 2023-10-30T12:12:13.691 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:19 24.1 24:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v8 -- 0x563c08b46e00 con 0x563c089c5aa0 2023-10-30T12:12:13.692 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:13.692 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.644+0000 7f6839c38780 5 client.4465.objecter 1 in flight 2023-10-30T12:12:13.692 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.684+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 22 ==== osd_map(41..43 src has 1..43) v4 ==== 3051+0+0 (crc 0 0 0) 0x7f6830005d80 con 0x563c089c5aa0 2023-10-30T12:12:13.692 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.684+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(41..43 src has 1..43) v4 2023-10-30T12:12:13.693 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.684+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [41,43] > 40 2023-10-30T12:12:13.693 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.684+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 41 2023-10-30T12:12:13.693 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 10 client.4465.objecter checking op 19 2023-10-30T12:12:13.693 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 41 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:13.693 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:13.694 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 42 2023-10-30T12:12:13.694 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 10 client.4465.objecter checking op 19 2023-10-30T12:12:13.694 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 42 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:13.694 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:13.694 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 43 2023-10-30T12:12:13.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 10 client.4465.objecter checking op 19 2023-10-30T12:12:13.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 43 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:13.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:13.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:13.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 10 monclient: _renew_subs 2023-10-30T12:12:13.696 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:12:13.696 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=44}) v3 -- 0x7f682c057020 con 0x563c08a019f0 2023-10-30T12:12:13.696 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:13.696 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.685+0000 7f681ffff640 20 client.4465.objecter 19 24.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-10-30T12:12:13.696 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 11 ==== osd_map(44..45 src has 1..45) v4 ==== 3117+0+0 (secure 0 0 0) 0x7f682c097e90 con 0x563c08a019f0 2023-10-30T12:12:13.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(44..45 src has 1..45) v4 2023-10-30T12:12:13.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [44,45] > 43 2023-10-30T12:12:13.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 44 2023-10-30T12:12:13.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 10 client.4465.objecter checking op 19 2023-10-30T12:12:13.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 44 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:13.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:15.333 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 45 2023-10-30T12:12:15.333 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 10 client.4465.objecter checking op 19 2023-10-30T12:12:15.334 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 45 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:15.334 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:15.334 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:15.334 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 10 monclient: _renew_subs 2023-10-30T12:12:15.335 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:12:15.335 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=46}) v3 -- 0x7f682c094d30 con 0x563c08a019f0 2023-10-30T12:12:15.335 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:15.335 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:13.688+0000 7f681ffff640 20 client.4465.objecter 19 24.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-10-30T12:12:15.335 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.294+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 12 ==== osd_map(46..46 src has 1..46) v4 ==== 2105+0+0 (secure 0 0 0) 0x7f682c097e90 con 0x563c08a019f0 2023-10-30T12:12:15.336 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.294+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(46..46 src has 1..46) v4 2023-10-30T12:12:15.336 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.294+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [46,46] > 45 2023-10-30T12:12:15.336 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.294+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 46 2023-10-30T12:12:15.336 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.294+0000 7f681ffff640 10 client.4465.objecter checking op 19 2023-10-30T12:12:15.336 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.294+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 46 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:15.337 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.294+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:15.337 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.294+0000 7f681ffff640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:15.337 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.294+0000 7f681ffff640 10 monclient: _renew_subs 2023-10-30T12:12:15.337 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.294+0000 7f681ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:12:15.337 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.294+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=47}) v3 -- 0x7f682c094ed0 con 0x563c08a019f0 2023-10-30T12:12:15.337 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.294+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:15.338 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.294+0000 7f681ffff640 20 client.4465.objecter 19 24.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-10-30T12:12:15.338 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.599+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:15.338 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.599+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:15.338 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.599+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:15.338 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:14.599+0000 7f681effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:15.339 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.330+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 13 ==== osd_map(47..47 src has 1..47) v4 ==== 676+0+0 (secure 0 0 0) 0x7f682c097e90 con 0x563c08a019f0 2023-10-30T12:12:16.602 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.330+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(47..47 src has 1..47) v4 2023-10-30T12:12:16.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.330+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [47,47] > 46 2023-10-30T12:12:16.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.330+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 47 2023-10-30T12:12:16.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.330+0000 7f681ffff640 10 client.4465.objecter checking op 19 2023-10-30T12:12:16.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.330+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 47 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:16.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.330+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:16.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.330+0000 7f681ffff640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:16.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.330+0000 7f681ffff640 10 monclient: _renew_subs 2023-10-30T12:12:16.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.330+0000 7f681ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:12:16.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.330+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=48}) v3 -- 0x7f682c0950a0 con 0x563c08a019f0 2023-10-30T12:12:16.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.330+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:16.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.330+0000 7f681ffff640 20 client.4465.objecter 19 24.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-10-30T12:12:16.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.599+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:16.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.599+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:16.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:15.599+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:16.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.129+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 14 ==== osd_map(48..48 src has 1..48) v4 ==== 340+0+0 (secure 0 0 0) 0x7f682c094780 con 0x563c08a019f0 2023-10-30T12:12:16.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.129+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(48..48 src has 1..48) v4 2023-10-30T12:12:16.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.129+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [48,48] > 47 2023-10-30T12:12:16.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.129+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 48 2023-10-30T12:12:16.607 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.129+0000 7f681ffff640 10 client.4465.objecter checking op 19 2023-10-30T12:12:16.607 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.129+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 48 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:16.607 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.129+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:16.607 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.129+0000 7f681ffff640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:16.607 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.129+0000 7f681ffff640 10 monclient: _renew_subs 2023-10-30T12:12:16.608 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.129+0000 7f681ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:12:16.608 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.129+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=49}) v3 -- 0x7f682c095240 con 0x563c08a019f0 2023-10-30T12:12:16.608 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.129+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:16.608 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.129+0000 7f681ffff640 20 client.4465.objecter 19 24.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-10-30T12:12:16.608 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.599+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:16.609 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.599+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:17.559 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.599+0000 7fpi_service_pp: "distro": "centos", 2023-10-30T12:12:17.560 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-10-30T12:12:17.560 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "distro_version": "9", 2023-10-30T12:12:17.560 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "foo": "bar", 2023-10-30T12:12:17.560 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "hostname": "smithi003", 2023-10-30T12:12:17.560 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-10-30T12:12:17.561 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "kernel_version": "5.14.0-378.el9.x86_64", 2023-10-30T12:12:17.561 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "mem_swap_kb": "0", 2023-10-30T12:12:17.561 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "mem_total_kb": "32494896", 2023-10-30T12:12:17.561 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "os": "Linux", 2023-10-30T12:12:17.561 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "this": "that" 2023-10-30T12:12:17.561 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: }, 2023-10-30T12:12:17.562 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: "task_status": {} 2023-10-30T12:12:17.562 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: } 2023-10-30T12:12:17.562 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: } 2023-10-30T12:12:17.562 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: } 2023-10-30T12:12:17.562 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: } 2023-10-30T12:12:17.563 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: } 2023-10-30T12:12:17.563 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: 2023-10-30T12:12:17.563 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: attempt 1 of 20 2023-10-30T12:12:17.563 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10480 ms) 2023-10-30T12:12:17.563 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (36047 ms total) 2023-10-30T12:12:17.564 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: 2023-10-30T12:12:17.564 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: [----------] Global test environment tear-down 2023-10-30T12:12:17.564 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (36047 ms total) 2023-10-30T12:12:17.564 INFO:tasks.workunit.client.0.smithi003.stdout: api_service_pp: [ PASSED ] 4 tests. 2023-10-30T12:12:18.240 INFO:tasks.workunit.client.0.smithi003.stdout:681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:18.240 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:16.696+0000 7f683918b640 10 client.4465.objecter tick 2023-10-30T12:12:18.241 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.293+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 15 ==== osd_map(49..49 src has 1..49) v4 ==== 1046+0+0 (secure 0 0 0) 0x7f682c0950a0 con 0x563c08a019f0 2023-10-30T12:12:18.241 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.293+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(49..49 src has 1..49) v4 2023-10-30T12:12:18.241 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.293+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [49,49] > 48 2023-10-30T12:12:18.241 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.293+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 49 2023-10-30T12:12:18.241 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.293+0000 7f681ffff640 10 client.4465.objecter checking op 19 2023-10-30T12:12:18.242 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.293+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 49 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:18.242 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.293+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:18.242 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.293+0000 7f681ffff640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:18.242 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.293+0000 7f681ffff640 10 monclient: _renew_subs 2023-10-30T12:12:18.243 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.293+0000 7f681ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:12:18.243 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.293+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=50}) v3 -- 0x7f682c095410 con 0x563c08a019f0 2023-10-30T12:12:18.243 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.293+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:18.243 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.293+0000 7f681ffff640 20 client.4465.objecter 19 24.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-10-30T12:12:18.243 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.601+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:18.244 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.601+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:18.244 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:17.601+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:18.244 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.238+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 16 ==== osd_map(50..50 src has 1..50) v4 ==== 1419+0+0 (secure 0 0 0) 0x7f682c0950a0 con 0x563c08a019f0 2023-10-30T12:12:18.244 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.238+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(50..50 src has 1..50) v4 2023-10-30T12:12:18.244 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.238+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [50,50] > 49 2023-10-30T12:12:18.245 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.238+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 50 2023-10-30T12:12:18.245 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.238+0000 7f681ffff640 10 client.4465.objecter checking op 19 2023-10-30T12:12:18.245 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.238+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 50 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:18.245 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.238+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:18.245 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.238+0000 7f681ffff640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:18.246 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.238+0000 7f681ffff640 10 monclient: _renew_subs 2023-10-30T12:12:18.246 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.238+0000 7f681ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:12:19.244 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.238+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=51}) v3 -- 0x7f682c0955e0 con 0x563c08a019f0 2023-10-30T12:12:19.245 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.238+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:19.245 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.238+0000 7f681ffff640 20 client.4465.objecter 19 24.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-10-30T12:12:19.245 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.601+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:19.245 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.601+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:19.246 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:18.601+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:19.246 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 23 ==== osd_op_reply(19 exist [writefull 0~5] v43'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f683001c040 con 0x563c089c5aa0 2023-10-30T12:12:19.246 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(19 exist [writefull 0~5] v43'6 uv6 ondisk = 0) v8 2023-10-30T12:12:19.246 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:19.247 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 19 ondisk uv 6 in 24.1 attempt 0 2023-10-30T12:12:19.247 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 0 2023-10-30T12:12:19.247 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 19 2023-10-30T12:12:19.247 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6837728640 15 client.4465.objecter _finish_op 19 2023-10-30T12:12:19.247 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:19.247 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 19 2023-10-30T12:12:19.248 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6837728640 5 client.4465.objecter 0 in flight 2023-10-30T12:12:19.248 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6837728640 20 librados: complete_aio_write 0x563c08b47280 2023-10-30T12:12:19.248 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (5599 ms) 2023-10-30T12:12:19.248 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2023-10-30T12:12:19.248 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6839c38780 20 librados: queue_aio_write 0x563c08916db0 completion 0x563c08b47ce0 write_seq 6 2023-10-30T12:12:19.249 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6839c38780 10 client.4465.objecter _op_submit op 0x563c08a84af0 2023-10-30T12:12:19.249 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6839c38780 20 client.4465.objecter _calc_target epoch 50 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:19.249 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6839c38780 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:19.249 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6839c38780 10 client.4465.objecter _calc_target raw pgid 24.d3a59201 -> actual 24.1 acting [1,2] primary 1 2023-10-30T12:12:19.249 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6839c38780 20 client.4465.objecter _get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:19.250 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6839c38780 10 client.4465.objecter _op_submit oid exist '@24' '@24' [writefull 0~5 in=5b] tid 20 osd.1 2023-10-30T12:12:19.250 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6839c38780 20 client.4465.objecter get_session s=0x563c089175f0 osd=1 3 2023-10-30T12:12:19.250 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6839c38780 15 client.4465.objecter _session_op_assign 1 20 2023-10-30T12:12:19.250 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6839c38780 15 client.4465.objecter _send_op 20 to 24.1 on osd.1 2023-10-30T12:12:19.250 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 --> v2:172.21.15.3:6804/1445650280 -- osd_op(unknown.0.0:20 24.1 24:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) v8 -- 0x563c08b46e00 con 0x563c089c5aa0 2023-10-30T12:12:20.491 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:20.492 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.243+0000 7f6839c38780 5 client.4465.objecter 1 in flight 2023-10-30T12:12:20.492 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.322+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 17 ==== osd_map(51..51 src has 1..51) v4 ==== 675+0+0 (secure 0 0 0) 0x7f682c094780 con 0x563c08a019f0 2023-10-30T12:12:20.492 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.322+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(51..51 src has 1..51) v4 2023-10-30T12:12:20.492 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.322+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [51,51] > 50 2023-10-30T12:12:20.493 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.322+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 51 2023-10-30T12:12:20.493 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.322+0000 7f681ffff640 10 client.4465.objecter checking op 20 2023-10-30T12:12:20.493 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.322+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 51 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:20.493 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.322+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:20.493 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.322+0000 7f681ffff640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:20.494 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.322+0000 7f681ffff640 10 monclient: _renew_subs 2023-10-30T12:12:20.494 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.322+0000 7f681ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:12:20.494 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.322+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=52}) v3 -- 0x7f682408bb10 con 0x563c08a019f0 2023-10-30T12:12:20.494 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.322+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:20.495 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.322+0000 7f681ffff640 20 client.4465.objecter 20 24.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-10-30T12:12:20.495 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.601+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:20.495 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.601+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:20.495 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:19.601+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:20.495 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.490+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 18 ==== osd_map(52..52 src has 1..52) v4 ==== 352+0+0 (secure 0 0 0) 0x7f682c052c20 con 0x563c08a019f0 2023-10-30T12:12:20.496 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.490+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(52..52 src has 1..52) v4 2023-10-30T12:12:20.496 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.490+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [52,52] > 51 2023-10-30T12:12:20.496 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.490+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 52 2023-10-30T12:12:20.496 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.490+0000 7f681ffff640 10 client.4465.objecter checking op 20 2023-10-30T12:12:20.496 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.490+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 52 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:20.497 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.490+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:20.497 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.490+0000 7f681ffff640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:20.497 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.490+0000 7f681ffff640 10 monclient: _renew_subs 2023-10-30T12:12:20.497 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.490+0000 7f681ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:12:20.497 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.490+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=53}) v3 -- 0x7f682407d820 con 0x563c08a019f0 2023-10-30T12:12:21.020 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:03:20.956 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:21.053 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:03:20.988 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:21.101 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:03:21.036 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:21.108 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:03:21.044 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:21.129 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:03:21.037 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:21.131 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:03:21.040 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:21.258 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:03:21.166 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:21.265 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:03:21.200 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:21.410 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:03:21.346 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:22.878 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.490+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:22.878 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.490+0000 7f681ffff640 20 client.4465.objecter 20 24.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-10-30T12:12:22.878 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.601+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:22.879 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.601+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:22.879 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:20.601+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:22.879 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:21.601+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:22.879 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:21.601+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:22.879 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:21.601+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:22.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:21.697+0000 7f683918b640 10 client.4465.objecter tick 2023-10-30T12:12:22.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.081+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 19 ==== osd_map(53..53 src has 1..53) v4 ==== 933+0+0 (secure 0 0 0) 0x7f682c094780 con 0x563c08a019f0 2023-10-30T12:12:22.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.081+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(53..53 src has 1..53) v4 2023-10-30T12:12:22.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.081+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [53,53] > 52 2023-10-30T12:12:22.881 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.081+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 53 2023-10-30T12:12:22.881 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.081+0000 7f681ffff640 10 client.4465.objecter checking op 20 2023-10-30T12:12:22.881 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.081+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 53 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:22.881 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.081+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:22.881 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.081+0000 7f681ffff640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:22.882 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.081+0000 7f681ffff640 10 monclient: _renew_subs 2023-10-30T12:12:22.882 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.081+0000 7f681ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:12:22.882 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.081+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=54}) v3 -- 0x7f682407ced0 con 0x563c08a019f0 2023-10-30T12:12:22.882 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.081+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:22.883 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.081+0000 7f681ffff640 20 client.4465.objecter 20 24.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-10-30T12:12:22.883 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.601+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:22.883 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.601+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:22.883 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.601+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:22.883 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.876+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 <== mon.1 v2:172.21.15.92:3300/0 20 ==== osd_map(54..54 src has 1..54) v4 ==== 4227+0+0 (secure 0 0 0) 0x7f682c094780 con 0x563c08a019f0 2023-10-30T12:12:22.884 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.876+0000 7f681ffff640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_map(54..54 src has 1..54) v4 2023-10-30T12:12:22.884 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.876+0000 7f681ffff640 3 client.4465.objecter handle_osd_map got epochs [54,54] > 53 2023-10-30T12:12:22.884 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.876+0000 7f681ffff640 3 client.4465.objecter handle_osd_map decoding incremental epoch 54 2023-10-30T12:12:22.884 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.876+0000 7f681ffff640 10 client.4465.objecter checking op 20 2023-10-30T12:12:23.674 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.876+0000 7f681ffff640 20 client.4465.objecter _calc_target epoch 54 base exist @24 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:12:23.675 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.876+0000 7f681ffff640 20 client.4465.objecter _calc_target target exist @24 -> pgid 24.d3a59201 2023-10-30T12:12:23.675 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.876+0000 7f681ffff640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:23.675 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.876+0000 7f681ffff640 10 monclient: _renew_subs 2023-10-30T12:12:23.675 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.876+0000 7f681ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:12:23.675 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.876+0000 7f681ffff640 1 -- 172.21.15.3:0/1402585138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=55}) v3 -- 0x7f6824074210 con 0x563c08a019f0 2023-10-30T12:12:23.676 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.876+0000 7f681ffff640 20 client.4465.objecter dump_active .. 0 homeless 2023-10-30T12:12:23.676 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:22.876+0000 7f681ffff640 20 client.4465.objecter 20 24.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2023-10-30T12:12:23.676 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.601+0000 7f681effd640 10 monclient: tick 2023-10-30T12:12:23.676 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.601+0000 7f681effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:23.676 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.601+0000 7f681effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:23.676 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.672+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 <== osd.1 v2:172.21.15.3:6804/1445650280 24 ==== osd_op_reply(20 exist [writefull 0~5] v50'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f683001c040 con 0x563c089c5aa0 2023-10-30T12:12:23.677 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.672+0000 7f6837728640 10 client.4465.objecter ms_dispatch 0x563c0894dd50 osd_op_reply(20 exist [writefull 0~5] v50'7 uv7 ondisk = 0) v8 2023-10-30T12:12:23.677 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.672+0000 7f6837728640 10 client.4465.objecter in handle_osd_op_reply 2023-10-30T12:12:23.677 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.672+0000 7f6837728640 7 client.4465.objecter handle_osd_op_reply 20 ondisk uv 7 in 24.1 attempt 0 2023-10-30T12:12:23.677 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.672+0000 7f6837728640 10 client.4465.objecter op 0 rval 0 len 0 2023-10-30T12:12:23.677 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.672+0000 7f6837728640 15 client.4465.objecter handle_osd_op_reply completed tid 20 2023-10-30T12:12:23.678 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.672+0000 7f6837728640 15 client.4465.objecter _finish_op 20 2023-10-30T12:12:23.678 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.672+0000 7f6837728640 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 4 2023-10-30T12:12:23.678 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.672+0000 7f6837728640 15 client.4465.objecter _session_op_remove 1 20 2023-10-30T12:12:23.678 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.672+0000 7f6837728640 5 client.4465.objecter 0 in flight 2023-10-30T12:12:23.678 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.672+0000 7f6837728640 20 librados: complete_aio_write 0x563c08b47ce0 2023-10-30T12:12:23.679 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (4429 ms) 2023-10-30T12:12:23.679 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 10 librados: watch_flush enter 2023-10-30T12:12:23.679 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 10 librados: watch_flush exit 2023-10-30T12:12:23.679 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 10 client.4465.objecter close_session for osd.1 2023-10-30T12:12:23.679 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 >> v2:172.21.15.3:6804/1445650280 conn(0x563c089c5aa0 msgr2=0x563c08b461e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:23.679 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 1 --2- 172.21.15.3:0/1402585138 >> v2:172.21.15.3:6804/1445650280 conn(0x563c089c5aa0 0x563c08b461e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:23.680 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 20 client.4465.objecter put_session s=0x563c089175f0 osd=1 1 2023-10-30T12:12:23.680 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 20 client.4465.objecter shutdown clearing up homeless session... 2023-10-30T12:12:23.680 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 10 client.4465.objecter successfully canceled tick 2023-10-30T12:12:23.680 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 >> v2:172.21.15.92:6800/1750155735 conn(0x7f682406aa00 msgr2=0x7f682406cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:23.681 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 1 --2- 172.21.15.3:0/1402585138 >> v2:172.21.15.92:6800/1750155735 conn(0x7f682406aa00 0x7f682406cec0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6828005330 tx=0x7f6828010000 comp rx=0 tx=0).stop 2023-10-30T12:12:23.681 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 10 monclient: shutdown 2023-10-30T12:12:23.681 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:23.681 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c08a019f0 msgr2=0x563c08a84070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:23.681 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 1 --2- 172.21.15.3:0/1402585138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c08a019f0 0x563c08a84070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f682c00a520 tx=0x7f682c04ab60 comp rx=0 tx=0).stop 2023-10-30T12:12:23.681 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6837728640 1 -- 172.21.15.3:0/1402585138 reap_dead start 2023-10-30T12:12:23.682 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 shutdown_connections 2023-10-30T12:12:23.682 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 >> 172.21.15.3:0/1402585138 conn(0x563c08a05890 msgr2=0x563c08a9d960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:23.682 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 shutdown_connections 2023-10-30T12:12:23.682 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 1 -- 172.21.15.3:0/1402585138 wait complete. 2023-10-30T12:12:23.682 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.673+0000 7f6839c38780 1 librados: shutdown 2023-10-30T12:12:23.683 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [----------] 12 tests from AsioRados (19403 ms total) 2023-10-30T12:12:23.683 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: 2023-10-30T12:12:23.683 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [----------] Global test environment tear-down 2023-10-30T12:12:23.683 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (42377 ms total) 2023-10-30T12:12:23.683 INFO:tasks.workunit.client.0.smithi003.stdout: api_asio: [ PASSED ] 12 tests. 2023-10-30T12:12:30.686 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: Running main() from gmock_main.cc 2023-10-30T12:12:30.687 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2023-10-30T12:12:30.687 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [----------] Global test environment set-up. 2023-10-30T12:12:30.687 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [----------] 9 tests from LibRadosPools 2023-10-30T12:12:30.687 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2023-10-30T12:12:30.688 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ OK ] LibRadosPools.PoolList (3959 ms) 2023-10-30T12:12:30.688 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2023-10-30T12:12:30.688 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (4181 ms) 2023-10-30T12:12:30.688 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2023-10-30T12:12:30.689 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3812 ms) 2023-10-30T12:12:30.689 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2023-10-30T12:12:30.689 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (4923 ms) 2023-10-30T12:12:30.689 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2023-10-30T12:12:30.689 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (7363 ms) 2023-10-30T12:12:30.690 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2023-10-30T12:12:30.690 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5423 ms) 2023-10-30T12:12:30.690 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2023-10-30T12:12:30.690 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5179 ms) 2023-10-30T12:12:30.690 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2023-10-30T12:12:30.691 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5784 ms) 2023-10-30T12:12:30.691 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2023-10-30T12:12:30.691 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8715 ms) 2023-10-30T12:12:30.691 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [----------] 9 tests from LibRadosPools (49339 ms total) 2023-10-30T12:12:30.691 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: 2023-10-30T12:12:30.692 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [----------] Global test environment tear-down 2023-10-30T12:12:30.692 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (49339 ms total) 2023-10-30T12:12:30.692 INFO:tasks.workunit.client.0.smithi003.stdout: api_pool: [ PASSED ] 9 tests. 2023-10-30T12:12:41.501 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:03:41.436 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:41.509 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:03:41.445 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:41.576 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:03:41.485 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:41.577 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:03:41.486 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:41.600 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:03:41.535 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:41.653 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:03:41.588 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:41.661 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:03:41.597 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:41.677 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:03:41.585 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:41.768 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:03:41.703 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:12:44.833 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2023-10-30T12:12:44.834 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [----------] Global test environment set-up. 2023-10-30T12:12:44.834 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2023-10-30T12:12:44.834 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2023-10-30T12:12:44.834 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2023-10-30T12:12:44.835 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-10-30T12:12:44.835 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: 2023-10-30T12:12:44.835 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2023-10-30T12:12:44.835 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2023-10-30T12:12:44.835 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: unable to get monitor info from DNS SRV with service name: 2023-10-30T12:11:41.658+0000 7ff836842780 -1 failed for service _ceph-mon._tcp 2023-10-30T12:12:44.836 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: ceph-mon 2023-10-30T12:12:44.836 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: 2023-10-30T12:11:41.659+0000 7ff836842780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2023-10-30T12:12:44.836 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (382 ms) 2023-10-30T12:12:44.836 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2023-10-30T12:12:44.836 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2006 ms) 2023-10-30T12:12:44.837 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2388 ms total) 2023-10-30T12:12:44.837 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: 2023-10-30T12:12:44.837 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2023-10-30T12:12:44.838 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2023-10-30T12:12:44.838 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607be5678c0 2023-10-30T12:12:44.839 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607bdab6440 2023-10-30T12:12:44.839 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607bdbaa400 2023-10-30T12:12:44.839 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607bdb710b0 2023-10-30T12:12:44.839 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607bdcdb5a0 2023-10-30T12:12:44.839 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607bdcdbfb0 2023-10-30T12:12:44.840 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607bdcdc9c0 2023-10-30T12:12:44.840 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607bdcdd3d0 2023-10-30T12:12:44.840 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607bdcddde0 2023-10-30T12:12:44.840 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607bdcde7f0 2023-10-30T12:12:44.840 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607bdcdf200 2023-10-30T12:12:44.840 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607bdcdfc10 2023-10-30T12:12:44.841 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607bdce0600 2023-10-30T12:12:44.841 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607bdce0fb0 2023-10-30T12:12:44.841 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607bdce19c0 2023-10-30T12:12:44.841 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 0x5607bdce23a0 2023-10-30T12:12:44.841 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: started 16 aios 2023-10-30T12:12:44.842 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607be5678c0 2023-10-30T12:12:44.842 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607bdab6440 2023-10-30T12:12:44.842 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607bdbaa400 2023-10-30T12:12:44.842 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607bdb710b0 2023-10-30T12:12:44.842 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607bdcdb5a0 2023-10-30T12:12:44.843 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607bdcdbfb0 2023-10-30T12:12:44.843 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607bdcdc9c0 2023-10-30T12:12:44.843 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607bdcdd3d0 2023-10-30T12:12:44.843 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607bdcddde0 2023-10-30T12:12:44.843 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607bdcde7f0 2023-10-30T12:12:44.843 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607bdcdf200 2023-10-30T12:12:44.844 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607bdcdfc10 2023-10-30T12:12:44.844 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607bdce0600 2023-10-30T12:12:44.844 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607bdce0fb0 2023-10-30T12:12:44.844 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607bdce19c0 2023-10-30T12:12:44.844 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: waiting 0x5607bdce23a0 2023-10-30T12:12:44.845 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: done. 2023-10-30T12:12:44.845 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (23129 ms) 2023-10-30T12:12:44.845 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (23129 ms total) 2023-10-30T12:12:44.845 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: 2023-10-30T12:12:44.845 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2023-10-30T12:12:44.846 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2023-10-30T12:12:44.846 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2023-10-30T12:12:44.846 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2023-10-30T12:12:44.846 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ OK ] LibRadosMisc.Exec (17126 ms) 2023-10-30T12:12:44.846 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2023-10-30T12:12:44.847 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (5705 ms) 2023-10-30T12:12:44.847 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2023-10-30T12:12:44.847 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (8401 ms) 2023-10-30T12:12:44.847 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2023-10-30T12:12:58.290 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ OK ] LibRa api_watch_notify_pp: Running main() from gmock_main.cc 2023-10-30T12:12:58.291 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2023-10-30T12:12:58.291 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2023-10-30T12:12:58.291 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2023-10-30T12:12:58.291 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2023-10-30T12:12:58.291 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: notify 2023-10-30T12:12:58.292 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (30609 ms) 2023-10-30T12:12:58.292 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2023-10-30T12:12:58.292 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (10115 ms) 2023-10-30T12:12:58.292 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (40724 ms total) 2023-10-30T12:12:58.292 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: 2023-10-30T12:12:58.293 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2023-10-30T12:12:58.293 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2023-10-30T12:12:58.293 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: notify 2023-10-30T12:12:58.293 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (6929 ms) 2023-10-30T12:12:58.293 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2023-10-30T12:12:58.294 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: notify 2023-10-30T12:12:58.294 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (12470 ms) 2023-10-30T12:12:58.294 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2023-10-30T12:12:58.294 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (2414 ms) 2023-10-30T12:12:58.294 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2023-10-30T12:12:58.295 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (257 ms) 2023-10-30T12:12:58.295 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2023-10-30T12:12:58.295 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: handle_notify cookie 94398259612160 notify_id 330712481796 notifier_gid 4675 2023-10-30T12:12:58.295 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (379 ms) 2023-10-30T12:12:58.295 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2023-10-30T12:12:58.296 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: handle_notify cookie 94398262029088 notify_id 330712481798 notifier_gid 4675 2023-10-30T12:12:58.296 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (380 ms) 2023-10-30T12:12:58.296 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2023-10-30T12:12:58.296 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: handle_notify cookie 94398262029088 notify_id 330712481800 notifier_gid 4675 2023-10-30T12:12:58.296 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (253 ms) 2023-10-30T12:12:58.297 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2023-10-30T12:12:58.297 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: handle_notify cookie 94398262034208 notify_id 330712481801 notifier_gid 4675 2023-10-30T12:12:58.297 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (354 ms) 2023-10-30T12:12:58.297 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2023-10-30T12:12:58.297 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: handle_notify cookie 94398262035248 notify_id 330712481802 notifier_gid 4675 2023-10-30T12:12:58.297 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (250 ms) 2023-10-30T12:12:58.298 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2023-10-30T12:12:58.298 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: handle_notify cookie 94398262035248 notify_id 330712481803 notifier_gid 4675 2023-10-30T12:12:58.298 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (305 ms) 2023-10-30T12:12:58.298 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2023-10-30T12:12:58.298 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: trying... 2023-10-30T12:12:58.299 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: handle_notify cookie 94398262035248 notify_id 330712481804 notifier_gid 4675 2023-10-30T12:12:58.299 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: timed out 2023-10-30T12:13:05.705 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: flush api_list: [==========] Running 11 tests from 3 test suites. 2023-10-30T12:13:05.705 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [----------] Global test environment set-up. 2023-10-30T12:13:05.705 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [----------] 7 tests from LibRadosList 2023-10-30T12:13:05.706 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2023-10-30T12:13:05.706 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ OK ] LibRadosList.ListObjects (71516 ms) 2023-10-30T12:13:05.706 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2023-10-30T12:13:05.706 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (1274 ms) 2023-10-30T12:13:05.706 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2023-10-30T12:13:05.707 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: myset foo1,foo2,foo3 2023-10-30T12:13:05.707 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo1 2023-10-30T12:13:05.707 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo2 2023-10-30T12:13:05.707 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo3 2023-10-30T12:13:05.707 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: myset foo1,foo4,foo5 2023-10-30T12:13:05.708 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo4 2023-10-30T12:13:05.708 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo5 2023-10-30T12:13:05.708 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo1 2023-10-30T12:13:05.708 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: myset foo6,foo7 2023-10-30T12:13:05.708 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo7 2023-10-30T12:13:05.709 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo6 2023-10-30T12:13:05.709 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-10-30T12:13:05.709 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: ns1:foo4 2023-10-30T12:13:05.709 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: ns1:foo5 2023-10-30T12:13:05.709 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: ns2:foo7 2023-10-30T12:13:05.710 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: ns2:foo6 2023-10-30T12:13:05.710 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: ns1:foo1 2023-10-30T12:13:05.710 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: :foo1 2023-10-30T12:13:05.710 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: :foo2 2023-10-30T12:13:05.710 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: :foo3 2023-10-30T12:13:05.710 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (2579 ms) 2023-10-30T12:13:05.711 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2023-10-30T12:13:05.711 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 1 0 2023-10-30T12:13:05.711 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 10 0 2023-10-30T12:13:05.711 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 13 0 2023-10-30T12:13:05.712 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 7 0 2023-10-30T12:13:05.712 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 14 0 2023-10-30T12:13:05.712 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 0 0 2023-10-30T12:13:05.712 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 15 0 2023-10-30T12:13:05.712 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 11 0 2023-10-30T12:13:05.713 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 5 0 2023-10-30T12:13:05.713 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 8 0 2023-10-30T12:13:05.713 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 6 0 2023-10-30T12:13:05.713 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 3 0 2023-10-30T12:13:05.713 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 4 0 2023-10-30T12:13:05.713 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 12 0 2023-10-30T12:13:05.714 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 9 0 2023-10-30T12:13:05.714 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 2 0 2023-10-30T12:13:05.714 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-10-30T12:13:05.714 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (2088 ms) 2023-10-30T12:13:05.714 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2023-10-30T12:13:05.715 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: x cursor=MIN 2023-10-30T12:13:05.715 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=1 cursor=4:02547ec2:::1:head 2023-10-30T12:13:05.715 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=10 cursor=4:52ea6a34:::10:head 2023-10-30T12:13:05.715 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=13 cursor=4:566253c9:::13:head 2023-10-30T12:13:05.715 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=7 cursor=4:5c6b0b28:::7:head 2023-10-30T12:13:05.716 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=14 cursor=4:62a1935d:::14:head 2023-10-30T12:13:05.716 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=0 cursor=4:6cac518f:::0:head 2023-10-30T12:13:05.716 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=15 cursor=4:863748b0:::15:head 2023-10-30T12:13:05.716 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=11 cursor=4:89d3ae78:::11:head 2023-10-30T12:13:05.716 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=5 cursor=4:b29083e3:::5:head 2023-10-30T12:13:05.716 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=8 cursor=4:bd63b0f1:::8:head 2023-10-30T12:13:05.717 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=6 cursor=4:c4fdafeb:::6:head 2023-10-30T12:13:05.717 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=3 cursor=4:cfc208b3:::3:head 2023-10-30T12:13:05.717 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=4 cursor=4:d83876eb:::4:head 2023-10-30T12:13:05.717 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=12 cursor=4:de5d7c5f:::12:head 2023-10-30T12:13:05.717 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=9 cursor=4:e960b815:::9:head 2023-10-30T12:13:05.718 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > oid=2 cursor=4:f905c69b:::2:head 2023-10-30T12:13:05.718 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: FIRST> seek to MIN oid=1 2023-10-30T12:13:05.718 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=1 cursor=4:02547ec2:::1:head 2023-10-30T12:13:05.718 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:02547ec2:::1:head 2023-10-30T12:13:05.718 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:02547ec2:::1:head -> 1 2023-10-30T12:13:05.718 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=10 cursor=4:52ea6a34:::10:head 2023-10-30T12:13:05.719 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:52ea6a34:::10:head 2023-10-30T12:13:05.719 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:52ea6a34:::10:head -> 10 2023-10-30T12:13:05.719 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=13 cursor=4:566253c9:::13:head 2023-10-30T12:13:05.719 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:566253c9:::13:head 2023-10-30T12:13:05.719 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:566253c9:::13:head -> 13 2023-10-30T12:13:05.720 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=7 cursor=4:5c6b0b28:::7:head 2023-10-30T12:13:05.720 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:5c6b0b28:::7:head 2023-10-30T12:13:05.720 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:5c6b0b28:::7:head -> 7 2023-10-30T12:13:06.612 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=14 cursor=4:62a1935d:::14: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2023-10-30T12:13:06.613 INFO:tasks.workunit.client.0.smithi003.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2023-10-30T12:13:06.613 INFO:tasks.workunit.client.0.smithi003.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2023-10-30T12:13:06.613 INFO:tasks.workunit.client.0.smithi003.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2023-10-30T12:13:06.613 INFO:tasks.workunit.client.0.smithi003.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (85060 ms) 2023-10-30T12:13:06.613 INFO:tasks.workunit.client.0.smithi003.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (85060 ms total) 2023-10-30T12:13:06.614 INFO:tasks.workunit.client.0.smithi003.stdout: api_cls_remote_reads: 2023-10-30T12:13:06.614 INFO:tasks.workunit.client.0.smithi003.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2023-10-30T12:13:06.614 INFO:tasks.workunit.client.0.smithi003.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (85060 ms total) 2023-10-30T12:13:06.614 INFO:tasks.workunit.client.0.smithi003.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2023-10-30T12:13:06.950 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: Running main() from gmock_main.cc 2023-10-30T12:13:06.950 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2023-10-30T12:13:06.951 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [----------] Global test environment set-up. 2023-10-30T12:13:06.951 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2023-10-30T12:13:06.951 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2023-10-30T12:13:06.951 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2023-10-30T12:13:06.951 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2023-10-30T12:13:06.952 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (14853 ms) 2023-10-30T12:13:06.952 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2023-10-30T12:13:06.952 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (12969 ms) 2023-10-30T12:13:06.952 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2023-10-30T12:13:06.952 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (37968 ms) 2023-10-30T12:13:06.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2023-10-30T12:13:06.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (688 ms) 2023-10-30T12:13:06.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2023-10-30T12:13:06.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (1041 ms) 2023-10-30T12:13:06.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2023-10-30T12:13:06.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (120 ms) 2023-10-30T12:13:06.954 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2023-10-30T12:13:06.954 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (2484 ms) 2023-10-30T12:13:06.954 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2023-10-30T12:13:06.954 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (162 ms) 2023-10-30T12:13:06.954 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2023-10-30T12:13:06.955 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (80 ms) 2023-10-30T12:13:06.955 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2023-10-30T12:13:06.955 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (245 ms) 2023-10-30T12:13:06.955 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2023-10-30T12:13:06.955 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (345 ms) 2023-10-30T12:13:06.956 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2023-10-30T12:13:06.956 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (137 ms) 2023-10-30T12:13:06.956 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2023-10-30T12:13:06.956 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (644 ms) 2023-10-30T12:13:06.956 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2023-10-30T12:13:06.956 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (389 ms) 2023-10-30T12:13:06.957 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2023-10-30T12:13:06.957 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (295 ms) 2023-10-30T12:13:06.957 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2023-10-30T12:13:06.957 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (135 ms) 2023-10-30T12:13:06.957 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (72555 ms total) 2023-10-30T12:13:06.958 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: 2023-10-30T12:13:06.958 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [----------] Global test environment tear-down 2023-10-30T12:13:06.958 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (85445 ms total) 2023-10-30T12:13:06.958 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2023-10-30T12:13:07.066 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd_pp: Running main() from gmock_main.cc 2023-10-30T12:13:07.066 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2023-10-30T12:13:07.067 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd_pp: [----------] Global test environment set-up. 2023-10-30T12:13:07.067 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2023-10-30T12:13:07.067 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2023-10-30T12:13:07.067 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (669 ms) 2023-10-30T12:13:07.067 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2023-10-30T12:13:07.068 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (168 ms) 2023-10-30T12:13:07.068 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2023-10-30T12:13:07.068 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (84787 ms) 2023-10-30T12:13:07.068 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (85624 ms total) 2023-10-30T12:13:07.068 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd_pp: 2023-10-30T12:13:07.069 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd_pp: [----------] Global test environment tear-down 2023-10-30T12:13:07.069 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (85624 ms total) 2023-10-30T12:13:07.069 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2023-10-30T12:13:14.860 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: Running main() from gmock_main.cc 2023-10-30T12:13:14.860 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2023-10-30T12:13:14.860 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: [----------] Global test environment set-up. 2023-10-30T12:13:14.861 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2023-10-30T12:13:14.861 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2023-10-30T12:13:14.861 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (821 ms) 2023-10-30T12:13:14.861 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2023-10-30T12:13:14.861 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (102 ms) 2023-10-30T12:13:14.862 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2023-10-30T12:13:14.862 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (83118 ms) 2023-10-30T12:13:14.862 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2023-10-30T12:13:14.862 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:12:56.180802+0000 mon.a [INF] from='client.? 172.21.15.3:0/4228368684' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi003-92123-4"}]: dispatch 2023-10-30T12:13:14.863 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:12:59.865610+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi003-92123-4"}]: dispatch 2023-10-30T12:13:14.863 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:00.502882+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi003-92518-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T12:13:14.863 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:00.508467+0000 mon.c [INF] from='client.4776 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi003-92518-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T12:13:14.863 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:04.863602+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi003-92123-4"}]': finished 2023-10-30T12:13:14.863 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:06.012174+0000 mon.a [INF] from='client.? 172.21.15.3:0/4228368684' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi003-92123-4", "tierpool": "test-rados-api-smithi003-92123-4-cache"}]: dispatch 2023-10-30T12:13:14.864 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:07.739060+0000 mon.c [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-30T12:13:14.864 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:07.759507+0000 client.admin [INF] onexx 2023-10-30T12:13:14.864 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:07.834274+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi003-92123-4", "tierpool": "test-rados-api-smithi003-92123-4-cache"}]: dispatch 2023-10-30T12:13:14.864 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:07.872616+0000 mon.c [INF] from='client.? 172.21.15.3:0/1846555940' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-10-30T12:13:14.864 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:08.586477+0000 mon.b [INF] from='client.? 172.21.15.3:0/1953164076' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi003-91934-22"}]: dispatch 2023-10-30T12:13:14.865 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:08.670227+0000 mon.b [INF] from='client.? 172.21.15.3:0/4198798151' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi003-91922-16"}]: dispatch 2023-10-30T12:13:14.865 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:09.763836+0000 mon.c [WRN] Health check update: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-10-30T12:13:14.865 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:09.922476+0000 mon.c [INF] from='client.? 172.21.15.3:0/3405616038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi003-91967-10"}]: dispatch 2023-10-30T12:13:14.865 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:09.998802+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi003-91934-22"}]: dispatch 2023-10-30T12:13:16.394 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T1head 2023-10-30T12:13:16.395 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:62a1935d:::14:head 2023-10-30T12:13:16.395 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:62a1935d:::14:head -> 14 2023-10-30T12:13:16.395 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=0 cursor=4:6cac518f:::0:head 2023-10-30T12:13:16.396 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:6cac518f:::0:head 2023-10-30T12:13:16.396 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:6cac518f:::0:head -> 0 2023-10-30T12:13:16.396 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=15 cursor=4:863748b0:::15:head 2023-10-30T12:13:16.396 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:863748b0:::15:head 2023-10-30T12:13:16.396 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:863748b0:::15:head -> 15 2023-10-30T12:13:16.397 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=11 cursor=4:89d3ae78:::11:head 2023-10-30T12:13:16.397 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:89d3ae78:::11:head 2023-10-30T12:13:16.397 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:89d3ae78:::11:head -> 11 2023-10-30T12:13:16.397 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=5 cursor=4:b29083e3:::5:head 2023-10-30T12:13:16.397 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:b29083e3:::5:head 2023-10-30T12:13:16.397 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:b29083e3:::5:head -> 5 2023-10-30T12:13:16.398 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=8 cursor=4:bd63b0f1:::8:head 2023-10-30T12:13:16.398 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:bd63b0f1:::8:head 2023-10-30T12:13:16.398 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:bd63b0f1:::8:head -> 8 2023-10-30T12:13:16.398 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=6 cursor=4:c4fdafeb:::6:head 2023-10-30T12:13:16.398 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:c4fdafeb:::6:head 2023-10-30T12:13:16.399 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:c4fdafeb:::6:head -> 6 2023-10-30T12:13:16.399 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=3 cursor=4:cfc208b3:::3:head 2023-10-30T12:13:16.399 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:cfc208b3:::3:head 2023-10-30T12:13:16.399 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:cfc208b3:::3:head -> 3 2023-10-30T12:13:16.399 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=4 cursor=4:d83876eb:::4:head 2023-10-30T12:13:16.400 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:d83876eb:::4:head 2023-10-30T12:13:16.400 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:d83876eb:::4:head -> 4 2023-10-30T12:13:16.400 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=12 cursor=4:de5d7c5f:::12:head 2023-10-30T12:13:16.400 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:de5d7c5f:::12:head 2023-10-30T12:13:16.400 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:de5d7c5f:::12:head -> 12 2023-10-30T12:13:16.401 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=9 cursor=4:e960b815:::9:head 2023-10-30T12:13:16.401 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:e960b815:::9:head 2023-10-30T12:13:16.401 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:e960b815:::9:head -> 9 2023-10-30T12:13:16.401 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : oid=2 cursor=4:f905c69b:::2:head 2023-10-30T12:13:16.402 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:f905c69b:::2:head 2023-10-30T12:13:16.402 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:f905c69b:::2:head -> 2 2023-10-30T12:13:16.402 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:c4fdafeb:::6:head 2023-10-30T12:13:16.402 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : cursor()=4:c4fdafeb:::6:head expected=4:c4fdafeb:::6:head 2023-10-30T12:13:16.402 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:c4fdafeb:::6:head -> 6 2023-10-30T12:13:16.403 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=6 expected=6 2023-10-30T12:13:16.403 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:52ea6a34:::10:head 2023-10-30T12:13:16.403 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : cursor()=4:52ea6a34:::10:head expected=4:52ea6a34:::10:head 2023-10-30T12:13:16.403 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:52ea6a34:::10:head -> 10 2023-10-30T12:13:16.403 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=10 expected=10 2023-10-30T12:13:16.404 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:02547ec2:::1:head 2023-10-30T12:13:16.404 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : cursor()=4:02547ec2:::1:head expected=4:02547ec2:::1:head 2023-10-30T12:13:16.404 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:02547ec2:::1:head -> 1 2023-10-30T12:13:16.404 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=1 expected=1 2023-10-30T12:13:16.404 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:566253c9:::13:head 2023-10-30T12:13:16.405 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : cursor()=4:566253c9:::13:head expected=4:566253c9:::13:head 2023-10-30T12:13:16.405 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:566253c9:::13:head -> 13 2023-10-30T12:13:16.405 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=13 expected=13 2023-10-30T12:13:16.405 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:6cac518f:::0:head 2023-10-30T12:13:16.405 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : cursor()=4:6cac518f:::0:head expected=4:6cac518f:::0:head 2023-10-30T12:13:16.406 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:6cac518f:::0:head -> 0 2023-10-30T12:13:16.406 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=0 expected=0 2023-10-30T12:13:16.406 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:62a1935d:::14:head 2023-10-30T12:13:16.406 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : cursor()=4:62a1935d:::14:head expected=4:62a1935d:::14:head 2023-10-30T12:13:16.407 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:62a1935d:::14:head -> 14 2023-10-30T12:13:16.407 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=14 expected=14 2023-10-30T12:13:16.407 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:cfc208b3:::3:head 2023-10-30T12:13:16.407 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : cursor()=4:cfc208b3:::3:head expected=4:cfc208b3:::3:head 2023-10-30T12:13:16.407 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:cfc208b3:::3:head -> 3 2023-10-30T12:13:16.408 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=3 expected=3 2023-10-30T12:13:16.408 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:bd63b0f1:::8:head 2023-10-30T12:13:16.408 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : cursor()=4:bd63b0f1:::8:head expected=4:bd63b0f1:::8:head 2023-10-30T12:13:16.408 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:bd63b0f1:::8:head -> 8 2023-10-30T12:13:16.408 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=8 expected=8 2023-10-30T12:13:16.409 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:b29083e3:::5:head 2023-10-30T12:13:22.329 INFO:tasks.ceph.mon.a.smithi003.stderr:2023-10-30T12:13:22.289+0000 fb6a640 -1 mon.a@0(peon).paxos(paxos updating c 1..350) lease_expire from mon.2 v2:172.21.15.3:3301/0 is 1.026368380s seconds in the past; mons are probably laggy (or possibly clocks are too skewed) 2023-10-30T12:13:30.128 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_1_[92641]: starting. 2023-10-30T12:13:30.129 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_1_[92641]: creating pool ceph_test_rados_list_parallel.smithi003-92603 2023-10-30T12:13:30.129 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_1_[92641]: created object 0... 2023-10-30T12:13:30.129 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_1_[92641]: created object 25... 2023-10-30T12:13:30.129 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_1_[92641]: created object 49... 2023-10-30T12:13:30.130 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_1_[92641]: finishing. 2023-10-30T12:13:30.130 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_1_[92641]: shutting down. 2023-10-30T12:13:30.130 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_2_[92643]: starting. 2023-10-30T12:13:30.130 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_2_[92643]: listing objects. 2023-10-30T12:13:30.131 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_2_[92643]: listed object 0... 2023-10-30T12:13:30.131 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_2_[92643]: listed object 25... 2023-10-30T12:13:30.131 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_2_[92643]: saw 50 objects 2023-10-30T12:13:30.131 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_2_[92643]: shutting down. 2023-10-30T12:13:30.131 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:30.132 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_3_[108723]: starting. 2023-10-30T12:13:30.132 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_3_[108723]: creating pool ceph_test_rados_list_parallel.smithi003-92603 2023-10-30T12:13:30.132 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_3_[108723]: created object 0... 2023-10-30T12:13:30.132 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_3_[108723]: created object 25... 2023-10-30T12:13:30.132 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_3_[108723]: created object 49... 2023-10-30T12:13:30.133 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_3_[108723]: finishing. 2023-10-30T12:13:30.133 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_3_[108723]: shutting down. 2023-10-30T12:13:30.133 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:30.133 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_4_[108724]: starting. 2023-10-30T12:13:30.133 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_4_[108724]: listing objects. 2023-10-30T12:13:30.134 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_4_[108724]: listed object 0... 2023-10-30T12:13:30.134 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_4_[108724]: listed object 25... 2023-10-30T12:13:30.134 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_4_[108724]: saw 47 objects 2023-10-30T12:13:30.134 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_4_[108724]: shutting down. 2023-10-30T12:13:30.134 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:30.135 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_5_[108725]: starting. 2023-10-30T12:13:30.135 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_5_[108725]: removed 25 objects... 2023-10-30T12:13:30.135 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_5_[108725]: removed half of the objects 2023-10-30T12:13:30.135 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_5_[108725]: removed 50 objects... 2023-10-30T12:13:30.135 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_5_[108725]: removed 50 objects 2023-10-30T12:13:30.136 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_5_[108725]: shutting down. 2023-10-30T12:13:30.136 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:30.136 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:30.136 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_6_[121333]: starting. 2023-10-30T12:13:30.136 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_6_[121333]: creating pool ceph_test_rados_list_parallel.smithi003-92603 2023-10-30T12:13:30.137 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_6_[121333]: created object 0... 2023-10-30T12:13:30.137 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_6_[121333]: created object 25... 2023-10-30T12:13:30.137 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_6_[121333]: created object 49... 2023-10-30T12:13:30.137 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_6_[121333]: finishing. 2023-10-30T12:13:30.138 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_6_[121333]: shutting down. 2023-10-30T12:13:30.138 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:30.138 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:30.138 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_7_[121334]: starting. 2023-10-30T12:13:30.139 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_7_[121334]: listing objects. 2023-10-30T12:13:30.139 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_7_[121334]: listed object 0... 2023-10-30T12:13:30.139 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_7_[121334]: listed object 25... 2023-10-30T12:13:30.139 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_7_[121334]: listed object 50... 2023-10-30T12:13:30.139 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_7_[121334]: saw 54 objects 2023-10-30T12:13:30.140 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_7_[121334]: shutting down. 2023-10-30T12:13:30.140 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:30.140 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:30.140 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_8_[121335]: starting. 2023-10-30T12:13:30.140 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_8_[121335]: added 25 objects... 2023-10-30T12:13:30.141 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_8_[121335]: added half of the objects 2023-10-30T12:13:30.141 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_8_[121335]: added 50 objects... 2023-10-30T12:13:30.141 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_8_[121335]: added 50 objects 2023-10-30T12:13:30.141 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_8_[121335]: shutting down. 2023-10-30T12:13:30.141 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:30.142 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:30.462 INFO:tasks.workunit.client.0.smithi003.stdout: 2:13:10.041184+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi003-91922-16"}]: dispatch 2023-10-30T12:13:30.463 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:10.613141+0000 mon.b [INF] from='client.? 172.21.15.3:0/789932710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi003-92066-10"}]: dispatch 2023-10-30T12:13:30.463 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:10.793792+0000 mon.b [INF] from='client.? 172.21.15.3:0/3954268552' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi003-92319-7"}]: dispatch 2023-10-30T12:13:30.463 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:11.581683+0000 mon.a [INF] from='client.? 172.21.15.3:0/4294248822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi003-92332-12"}]: dispatch 2023-10-30T12:13:30.463 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:12.392734+0000 mon.b [INF] from='client.? 172.21.15.3:0/1104937471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi003-92259-7"}]: dispatch 2023-10-30T12:13:30.464 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:12.847315+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi003-92332-12"}]: dispatch 2023-10-30T12:13:30.464 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:12.862705+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi003-92123-4", "tierpool": "test-rados-api-smithi003-92123-4-cache"}]': finished 2023-10-30T12:13:30.464 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:13.137046+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi003-92066-10"}]: dispatch 2023-10-30T12:13:30.464 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:13.143635+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi003-92319-7"}]: dispatch 2023-10-30T12:13:30.464 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:13.306763+0000 mon.b [INF] from='client.? 172.21.15.3:0/3563866330' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi003-92367-4"}]: dispatch 2023-10-30T12:13:30.465 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:13.587849+0000 mon.c [INF] from='client.? 172.21.15.3:0/3405616038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi003-91967-10"}]: dispatch 2023-10-30T12:13:30.465 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:14.347158+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi003-92259-7"}]: dispatch 2023-10-30T12:13:30.465 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:14.411444+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi003-92367-4"}]: dispatch 2023-10-30T12:13:30.465 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:14.510826+0000 mon.c [INF] from='client.? 172.21.15.3:0/1846555940' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2023-10-30T12:13:30.465 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:14.925431+0000 mon.c [INF] from='client.? 172.21.15.3:0/3405616038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi003-91967-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:13:30.466 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:15.237180+0000 client.admin [INF] twoxx 2023-10-30T12:13:30.466 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:15.252556+0000 mon.c [INF] from='client.? 172.21.15.3:0/1846555940' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-10-30T12:13:39.486 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:14.097377+0000 mon.b [INF] from='client.? 172.21.15.3:0/1953164076' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi003-91934-22"}]: dispatch 2023-10-30T12:13:39.486 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:14.103258+0000 mon.b [INF] from='client.? 172.21.15.3:0/4198798151' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi003-91922-16"}]: dispatch 2023-10-30T12:13:39.487 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:14.442648+0000 mon.a [INF] from='client.? 172.21.15.3:0/4294248822' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi003-92332-12"}]: dispatch 2023-10-30T12:13:39.487 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:15.233042+0000 mon.a [INF] from='client.? 172.21.15.3:0/3106802346' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi003-91875-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:13:39.487 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:15.496635+0000 mon.c [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-30T12:13:39.487 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:15.959594+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi003-92332-12"}]: dispatch 2023-10-30T12:13:39.487 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:15.994993+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi003-91875-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:13:39.488 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:16.170234+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi003-92066-10"}]': finished 2023-10-30T12:13:39.488 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:16.178778+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi003-92319-7"}]': finished 2023-10-30T12:13:39.488 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:16.333321+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi003-91934-22"}]: dispatch 2023-10-30T12:13:39.488 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:16.345996+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi003-91922-16"}]: dispatch 2023-10-30T12:13:39.488 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:21.421064+0000 mon.c [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-30T12:13:39.489 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:23.134240+0000 mon.c [INF] from='client.? 172.21.15.3:0/1846555940' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-10-30T12:13:39.489 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:17.487125+0000 mon.b [INF] from='client.? 172.21.15.3:0/3954268552' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi003-92319-7"}]: dispatch 2023-10-30T12:13:39.489 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:17.503060+0000 mon.b [INF] from='client.? 172.21.15.3:0/789932710' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi003-92066-10"}]: dispatch 2023-10-30T12:13:39.489 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:17.522885+0000 mon.b [INF] from='client.? 172.21.15.3:0/1104937471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi003-92259-7"}]: dispatch 2023-10-30T12:13:39.489 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:17.525490+0000 mon.b [INF] from='client.? 172.21.15.3:0/4198798151' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi003-91922-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:13:39.489 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:17.528072+0000 mon.b [INF] from='client.? 172.21.15.3:0/1953164076' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi003-91934-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:13:40.230 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:22.648814+0000 mon.a [INF] from='client.? list_parallel: ******************************* 2023-10-30T12:13:40.230 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_9_[128522]: starting. 2023-10-30T12:13:40.230 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_9_[128522]: creating pool ceph_test_rados_list_parallel.smithi003-92603 2023-10-30T12:13:40.231 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_9_[128522]: created object 0... 2023-10-30T12:13:40.231 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_9_[128522]: created object 25... 2023-10-30T12:13:40.231 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_9_[128522]: created object 49... 2023-10-30T12:13:40.231 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_9_[128522]: finishing. 2023-10-30T12:13:40.231 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_9_[128522]: shutting down. 2023-10-30T12:13:40.232 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.232 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.232 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.232 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_10_[128523]: starting. 2023-10-30T12:13:40.232 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_10_[128523]: listing objects. 2023-10-30T12:13:40.232 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_10_[128523]: listed object 0... 2023-10-30T12:13:40.233 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_10_[128523]: listed object 25... 2023-10-30T12:13:40.233 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_10_[128523]: listed object 50... 2023-10-30T12:13:40.233 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_10_[128523]: listed object 75... 2023-10-30T12:13:40.233 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_10_[128523]: saw 98 objects 2023-10-30T12:13:40.233 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_10_[128523]: shutting down. 2023-10-30T12:13:40.233 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.234 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.234 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.234 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_13_[128526]: starting. 2023-10-30T12:13:40.234 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_13_[128526]: removed 25 objects... 2023-10-30T12:13:40.234 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_13_[128526]: removed half of the objects 2023-10-30T12:13:40.235 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_13_[128526]: removed 50 objects... 2023-10-30T12:13:40.235 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_13_[128526]: removed 50 objects 2023-10-30T12:13:40.235 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_13_[128526]: shutting down. 2023-10-30T12:13:40.235 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.235 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.235 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.236 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_12_[128525]: starting. 2023-10-30T12:13:40.236 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_12_[128525]: added 25 objects... 2023-10-30T12:13:40.236 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_12_[128525]: added half of the objects 2023-10-30T12:13:40.236 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_12_[128525]: added 50 objects... 2023-10-30T12:13:40.236 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_12_[128525]: added 50 objects 2023-10-30T12:13:40.236 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_12_[128525]: shutting down. 2023-10-30T12:13:40.237 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.237 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.237 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.237 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_11_[128524]: starting. 2023-10-30T12:13:40.237 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_11_[128524]: added 25 objects... 2023-10-30T12:13:40.237 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_11_[128524]: added half of the objects 2023-10-30T12:13:40.238 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_11_[128524]: added 50 objects... 2023-10-30T12:13:40.238 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_11_[128524]: added 50 objects 2023-10-30T12:13:40.238 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_11_[128524]: shutting down. 2023-10-30T12:13:40.238 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.238 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.239 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.239 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.239 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_14_[138835]: starting. 2023-10-30T12:13:40.239 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_14_[138835]: creating pool ceph_test_rados_list_parallel.smithi003-92603 2023-10-30T12:13:40.239 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_14_[138835]: created object 0... 2023-10-30T12:13:40.240 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_14_[138835]: created object 25... 2023-10-30T12:13:40.240 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_14_[138835]: created object 49... 2023-10-30T12:13:40.240 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_14_[138835]: finishing. 2023-10-30T12:13:40.240 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_14_[138835]: shutting down. 2023-10-30T12:13:40.240 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.240 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.241 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.241 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:40.241 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_15_[138836]: starting. 2023-10-30T12:13:40.241 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_15_[138836]: listing objects. 2023-10-30T12:13:41.416 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_15_[138836]: api_lock_pp: Running main() from gmock_main.cc 2023-10-30T12:13:41.417 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2023-10-30T12:13:41.417 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [----------] Global test environment set-up. 2023-10-30T12:13:41.417 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2023-10-30T12:13:41.417 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: seed 92066 2023-10-30T12:13:41.418 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2023-10-30T12:13:41.418 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (19610 ms) 2023-10-30T12:13:41.418 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2023-10-30T12:13:41.418 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (231 ms) 2023-10-30T12:13:41.419 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2023-10-30T12:13:41.419 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (5797 ms) 2023-10-30T12:13:41.419 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2023-10-30T12:13:41.419 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (8913 ms) 2023-10-30T12:13:41.419 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2023-10-30T12:13:41.419 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (3921 ms) 2023-10-30T12:13:41.420 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2023-10-30T12:13:41.420 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (6833 ms) 2023-10-30T12:13:41.420 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2023-10-30T12:13:41.420 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (5865 ms) 2023-10-30T12:13:41.420 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2023-10-30T12:13:41.421 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (4668 ms) 2023-10-30T12:13:41.421 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (55838 ms total) 2023-10-30T12:13:41.421 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: 2023-10-30T12:13:41.421 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2023-10-30T12:13:41.421 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2023-10-30T12:13:41.422 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (9347 ms) 2023-10-30T12:13:41.422 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2023-10-30T12:13:41.422 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (2766 ms) 2023-10-30T12:13:41.422 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2023-10-30T12:13:41.422 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1247 ms) 2023-10-30T12:13:41.423 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2023-10-30T12:13:41.423 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1329 ms) 2023-10-30T12:13:41.423 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2023-10-30T12:13:41.423 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (183 ms) 2023-10-30T12:13:41.423 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2023-10-30T12:13:41.424 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (246 ms) 2023-10-30T12:13:41.424 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2023-10-30T12:13:41.424 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (319 ms) 2023-10-30T12:13:41.424 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2023-10-30T12:13:41.424 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (215 ms) 2023-10-30T12:13:41.424 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (15652 ms total) 2023-10-30T12:13:41.425 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: 2023-10-30T12:13:41.425 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [----------] Global test environment tear-down 2023-10-30T12:13:41.425 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (120136 ms total) 2023-10-30T12:13:41.425 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock_pp: [ PASSED ] 16 tests. 2023-10-30T12:13:41.425 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: Running main() from gmock_main.cc 2023-10-30T12:13:41.426 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2023-10-30T12:13:41.426 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [----------] Global test environment set-up. 2023-10-30T12:13:41.426 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2023-10-30T12:13:41.426 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: seed 92319 2023-10-30T12:13:41.426 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2023-10-30T12:13:41.427 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (15043 ms) 2023-10-30T12:13:41.427 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2023-10-30T12:13:41.427 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (6400 ms) 2023-10-30T12:13:41.427 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2023-10-30T12:13:41.427 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (18 ms) 2023-10-30T12:13:41.428 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2023-10-30T12:13:41.428 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (3585 ms) 2023-10-30T12:13:41.428 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2023-10-30T12:13:41.428 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (8860 ms) 2023-10-30T12:13:41.428 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (33907 ms total) 2023-10-30T12:13:41.429 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: 2023-10-30T12:13:41.429 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2023-10-30T12:13:41.429 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2023-10-30T12:13:41.429 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (10627 ms) 2023-10-30T12:13:41.429 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2023-10-30T12:13:41.429 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (87 ms) 2023-10-30T12:13:41.430 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2023-10-30T12:13:41.430 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (24860 ms) 2023-10-30T12:13:41.430 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2023-10-30T12:13:41.430 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (329 ms) 2023-10-30T12:13:41.430 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (35903 ms total) 2023-10-30T12:13:41.431 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: 2023-10-30T12:13:41.431 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [----------] Global test environment tear-down 2023-10-30T12:13:41.431 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (120013 ms total) 2023-10-30T12:13:41.431 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat_pp: [ PASSED ] 9 tests. 2023-10-30T12:13:52.489 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:04:52.422 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:13:52.522 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:04:52.457 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:13:52.523 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:04:52.431 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:13:52.525 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:04:52.434 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:13:52.613 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:04:52.548 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:13:52.665 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:04:52.573 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:13:52.668 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:04:52.604 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:13:52.676 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:04:52.612 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:13:52.749 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:04:52.684 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:13:53.895 INFO:tasks.workunit.client.0.smithi003.stdout: listed object 0... 2023-10-30T12:13:53.896 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_15_[138836]: listed object 25... 2023-10-30T12:13:53.896 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_15_[138836]: listed object 50... 2023-10-30T12:13:53.896 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_15_[138836]: listed object 75... 2023-10-30T12:13:53.896 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_15_[138836]: listed object 100... 2023-10-30T12:13:53.897 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_15_[138836]: listed object 125... 2023-10-30T12:13:53.897 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_15_[138836]: saw 150 objects 2023-10-30T12:13:53.897 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_15_[138836]: shutting down. 2023-10-30T12:13:53.897 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:53.897 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:53.898 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:53.898 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:53.898 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_16_[138837]: starting. 2023-10-30T12:13:53.898 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_16_[138837]: added 25 objects... 2023-10-30T12:13:53.898 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_16_[138837]: added half of the objects 2023-10-30T12:13:53.899 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_16_[138837]: added 50 objects... 2023-10-30T12:13:53.899 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_16_[138837]: added 50 objects 2023-10-30T12:13:53.899 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: process_16_[138837]: shutting down. 2023-10-30T12:13:53.899 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:53.899 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:53.900 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:53.900 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:53.900 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******************************* 2023-10-30T12:13:53.900 INFO:tasks.workunit.client.0.smithi003.stdout: list_parallel: ******* SUCCESS ********** 2023-10-30T12:13:54.602 INFO:tasks.workunit.client.0.smithi003.stdout:172.21.15.3:0/4294248822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi003-92332-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:13:54.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:24.730468+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi003-92319-7"}]: dispatch 2023-10-30T12:13:54.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:24.740709+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi003-92066-10"}]: dispatch 2023-10-30T12:13:54.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:24.744405+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi003-92259-7"}]: dispatch 2023-10-30T12:13:54.603 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:24.748636+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi003-91922-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:13:54.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:24.753938+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi003-91934-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:13:54.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:24.904369+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi003-92367-4"}]': finished 2023-10-30T12:13:54.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:24.929282+0000 mon.c [INF] from='client.? 172.21.15.3:0/3405616038' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi003-91967-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-10-30T12:13:54.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:24.931277+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi003-91875-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T12:13:54.604 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:25.202330+0000 mon.c [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi003-92332-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:13:54.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:25.629886+0000 mon.c [INF] from='client.? 172.21.15.3:0/3405616038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi003-91967-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi003-91967-10"}]: dispatch 2023-10-30T12:13:54.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:25.867631+0000 client.admin [INF] threexx 2023-10-30T12:13:54.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:25.887267+0000 mon.c [INF] from='client.? 172.21.15.3:0/1846555940' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-10-30T12:13:54.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: got: 2023-10-30T12:13:26.438049+0000 mon.c [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-30T12:13:54.605 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (49128 ms) 2023-10-30T12:13:54.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (133169 ms total) 2023-10-30T12:13:54.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: 2023-10-30T12:13:54.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: [----------] Global test environment tear-down 2023-10-30T12:13:54.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (133169 ms total) 2023-10-30T12:13:54.606 INFO:tasks.workunit.client.0.smithi003.stdout: api_cmd: [ PASSED ] 4 tests. 2023-10-30T12:13:54.689 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_1_[92671]: starting. 2023-10-30T12:13:54.690 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_1_[92671]: creating pool ceph_test_rados_open_pools_parallel.smithi003-92619 2023-10-30T12:13:54.690 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_1_[92671]: created object 0... 2023-10-30T12:13:54.690 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_1_[92671]: created object 25... 2023-10-30T12:13:54.690 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_1_[92671]: created object 49... 2023-10-30T12:13:54.691 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_1_[92671]: finishing. 2023-10-30T12:13:54.691 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_1_[92671]: shutting down. 2023-10-30T12:13:54.691 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_2_[92672]: starting. 2023-10-30T12:13:54.691 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_2_[92672]: rados_pool_create. 2023-10-30T12:13:54.691 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_2_[92672]: rados_ioctx_create. 2023-10-30T12:13:54.692 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_2_[92672]: shutting down. 2023-10-30T12:13:54.692 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: ******************************* 2023-10-30T12:13:54.692 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_3_[116314]: starting. 2023-10-30T12:13:54.692 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_3_[116314]: creating pool ceph_test_rados_open_pools_parallel.smithi003-92619 2023-10-30T12:13:54.692 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_3_[116314]: created object 0... 2023-10-30T12:13:54.693 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_3_[116314]: created object 25... 2023-10-30T12:13:54.693 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_3_[116314]: created object 49... 2023-10-30T12:13:54.693 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_3_[116314]: finishing. 2023-10-30T12:13:54.693 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_3_[116314]: shutting down. 2023-10-30T12:13:54.693 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: ******************************* 2023-10-30T12:13:54.694 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_4_[116315]: starting. 2023-10-30T12:13:54.694 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_4_[116315]: rados_pool_create. 2023-10-30T12:13:54.694 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_4_[116315]: rados_ioctx_create. 2023-10-30T12:13:54.694 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: process_4_[116315]: shutting down. 2023-10-30T12:13:54.694 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: ******************************* 2023-10-30T12:13:54.695 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: ******************************* 2023-10-30T12:13:54.695 INFO:tasks.workunit.client.0.smithi003.stdout: open_pools_parallel: ******* SUCCESS ********** 2023-10-30T12:13:54.702 INFO:tasks.workunit.client.0.smithi003.stdout:dosMisc.Applications (4696 ms) 2023-10-30T12:13:54.703 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2023-10-30T12:13:54.703 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2023-10-30T12:13:54.703 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2023-10-30T12:13:54.704 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2023-10-30T12:13:54.704 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2023-10-30T12:13:54.704 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (68259 ms) 2023-10-30T12:13:54.705 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [----------] 8 tests from LibRadosMisc (104187 ms total) 2023-10-30T12:13:54.705 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: 2023-10-30T12:13:54.705 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [----------] Global test environment tear-down 2023-10-30T12:13:54.705 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (133414 ms total) 2023-10-30T12:13:54.706 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc: [ PASSED ] 12 tests. 2023-10-30T12:13:54.717 INFO:tasks.workunit.client.0.smithi003.stdout:ing 2023-10-30T12:13:54.717 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: flushed 2023-10-30T12:13:54.717 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3237 ms) 2023-10-30T12:13:54.718 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2023-10-30T12:13:54.718 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: trying... 2023-10-30T12:13:54.718 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: handle_notify cookie 94398262017888 notify_id 330712481805 notifier_gid 4675 2023-10-30T12:13:54.718 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: timed out 2023-10-30T12:13:54.718 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: flushing 2023-10-30T12:13:54.719 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: flushed 2023-10-30T12:13:54.719 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3167 ms) 2023-10-30T12:13:54.719 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2023-10-30T12:13:54.719 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: List watches 2023-10-30T12:13:54.719 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: notify2 2023-10-30T12:13:54.720 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: handle_notify cookie 94398262018928 notify_id 335007449092 notifier_gid 4675 2023-10-30T12:13:54.720 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: notify2 done 2023-10-30T12:13:54.720 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: watch_check 2023-10-30T12:13:54.720 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: unwatch2 2023-10-30T12:13:54.720 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: flushing 2023-10-30T12:13:54.721 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: done 2023-10-30T12:13:54.721 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (5643 ms) 2023-10-30T12:13:54.721 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2023-10-30T12:13:54.721 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: List watches 2023-10-30T12:13:54.721 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: notify2 2023-10-30T12:13:54.721 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: handle_notify cookie 94398262018928 notify_id 339302416398 notifier_gid 4675 2023-10-30T12:13:54.722 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: notify2 done 2023-10-30T12:13:54.722 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: watch_check 2023-10-30T12:13:54.722 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: unwatch2 2023-10-30T12:13:54.722 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: flushing 2023-10-30T12:13:54.723 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: done 2023-10-30T12:13:54.723 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (4009 ms) 2023-10-30T12:13:54.723 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (40047 ms total) 2023-10-30T12:13:54.723 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: 2023-10-30T12:13:54.724 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2023-10-30T12:13:54.724 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (133308 ms total) 2023-10-30T12:13:54.724 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2023-10-30T12:13:54.725 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_1_[92677]: starting. 2023-10-30T12:13:54.725 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_1_[92677]: creating pool ceph_test_rados_delete_pools_parallel.smithi003-92668 2023-10-30T12:13:54.725 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_1_[92677]: created object 0... 2023-10-30T12:13:54.725 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_1_[92677]: created object 25... 2023-10-30T12:13:54.725 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_1_[92677]: created object 49... 2023-10-30T12:13:54.726 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_1_[92677]: finishing. 2023-10-30T12:13:54.726 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_1_[92677]: shutting down. 2023-10-30T12:13:54.726 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_2_[92679]: starting. 2023-10-30T12:13:54.726 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_2_[92679]: deleting pool ceph_test_rados_delete_pools_parallel.smithi003-92668 2023-10-30T12:13:54.727 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_2_[92679]: shutting down. 2023-10-30T12:13:54.727 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: ******************************* 2023-10-30T12:13:54.727 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_3_[116297]: starting. 2023-10-30T12:13:54.727 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_3_[116297]: creating pool ceph_test_rados_delete_pools_parallel.smithi003-92668 2023-10-30T12:13:54.727 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_3_[116297]: created object 0... 2023-10-30T12:13:54.728 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_3_[116297]: created object 25... 2023-10-30T12:13:54.728 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_3_[116297]: created object 49... 2023-10-30T12:13:54.728 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_3_[116297]: finishing. 2023-10-30T12:13:54.728 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_3_[116297]: shutting down. 2023-10-30T12:13:54.728 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: ******************************* 2023-10-30T12:13:54.729 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_5_[116307]: starting. 2023-10-30T12:13:54.729 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_5_[116307]: listing objects. 2023-10-30T12:13:54.729 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_5_[116307]: listed object 0... 2023-10-30T12:13:54.729 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_5_[116307]: listed object 25... 2023-10-30T12:13:54.729 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_5_[116307]: saw 50 objects 2023-10-30T12:13:54.729 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_5_[116307]: shutting down. 2023-10-30T12:13:54.730 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: ******************************* 2023-10-30T12:13:54.730 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_4_[116306]: starting. 2023-10-30T12:13:54.730 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_4_[116306]: deleting pool ceph_test_rados_delete_pools_parallel.smithi003-92668 2023-10-30T12:13:54.731 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: process_4_[116306]: shutting down. 2023-10-30T12:13:54.731 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: ******************************* 2023-10-30T12:13:54.731 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: ******************************* 2023-10-30T12:13:54.731 INFO:tasks.workunit.client.0.smithi003.stdout: delete_pools_parallel: ******* SUCCESS ********** 2023-10-30T12:14:07.788 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: Running main() from gmock_main.cc 2023-10-30T12:14:07.789 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2023-10-30T12:14:07.789 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [----------] Global test environment set-up. 2023-10-30T12:14:07.790 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2023-10-30T12:14:07.790 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2023-10-30T12:14:07.791 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: watch_notify_test_cb 2023-10-30T12:14:07.791 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: watch_notify_test_cb 2023-10-30T12:14:07.791 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (19737 ms) 2023-10-30T12:14:07.791 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2023-10-30T12:14:07.792 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-10-30T12:14:07.792 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94485447893440 err -107 2023-10-30T12:14:07.792 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (2894 ms) 2023-10-30T12:14:07.792 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2023-10-30T12:14:07.792 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-10-30T12:14:07.793 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94485447893440 err -107 2023-10-30T12:14:07.793 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (13870 ms) 2023-10-30T12:14:07.793 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2023-10-30T12:14:07.793 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: watch_notify2_test_cb from 4352 notify_id 274877906944 cookie 94485449411536 2023-10-30T12:14:07.793 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (27599 ms) 2023-10-30T12:14:07.794 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2023-10-30T12:14:07.794 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: watch_notify2_test_cb from 4352 notify_id 322122547201 cookie 94485449411536 2023-10-30T12:14:07.794 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (4761 ms) 2023-10-30T12:14:07.794 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2023-10-30T12:14:07.794 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: watch_notify2_test_cb from 4352 notify_id 330712481794 cookie 94485449415792 2023-10-30T12:14:07.795 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (921 ms) 2023-10-30T12:14:07.795 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2023-10-30T12:14:07.795 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: watch_notify2_test_cb from 4352 notify_id 330712481793 cookie 94485449451824 2023-10-30T12:14:07.795 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: watch_notify2_test_cb from 4352 notify_id 330712481793 cookie 94485449452864 2023-10-30T12:14:07.795 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (409 ms) 2023-10-30T12:14:07.796 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2023-10-30T12:14:07.796 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: watch_notify2_test_cb from 4352 notify_id 330712481795 cookie 94485449452864 2023-10-30T12:14:07.796 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: watch_notify2_test_cb from 4352 notify_id 330712481799 cookie 94485449452864 2023-10-30T12:14:07.796 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3586 ms) 2023-10-30T12:14:07.796 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2023-10-30T12:14:07.797 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2023-10-30T12:14:07.797 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94485449452864 err -107 2023-10-30T12:14:07.797 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: watch_notify2_test_cb from 4352 notify_id 335007449090 cookie 94485449452864 2023-10-30T12:14:07.797 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (6211 ms) 2023-10-30T12:14:07.797 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2023-10-30T12:14:07.798 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2023-10-30T12:14:07.798 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94485449452864 err -107 2023-10-30T12:14:07.798 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1157 ms) 2023-10-30T12:14:07.798 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (81145 ms total) 2023-10-30T12:14:07.798 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: 2023-10-30T12:14:07.799 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2023-10-30T12:14:07.799 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2023-10-30T12:14:07.799 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: watch_notify_test_cb 2023-10-30T12:14:07.799 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (10921 ms) 2023-10-30T12:14:07.799 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (10921 ms total) 2023-10-30T12:14:07.800 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: 2023-10-30T12:14:07.800 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [----------] Global test environment tear-down 2023-10-30T12:14:07.800 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (146386 ms total) 2023-10-30T12:14:07.801 INFO:tasks.workunit.client.0.smithi003.stdout: api_watch_notify: [ PASSED ] 11 tests. 2023-10-30T12:14:09.929 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: Running main() from gmock_main.cc 2023-10-30T12:14:09.929 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2023-10-30T12:14:09.930 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [----------] Global test environment set-up. 2023-10-30T12:14:09.930 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2023-10-30T12:14:09.930 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: seed 91934 2023-10-30T12:14:09.930 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2023-10-30T12:14:09.930 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2023-10-30T12:14:09.931 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2023-10-30T12:14:09.931 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (12402 ms) 2023-10-30T12:14:09.931 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2023-10-30T12:14:09.931 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (6811 ms) 2023-10-30T12:14:09.931 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2023-10-30T12:14:09.932 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (1073 ms) 2023-10-30T12:14:09.932 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2023-10-30T12:14:09.932 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (172 ms) 2023-10-30T12:14:09.932 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2023-10-30T12:14:09.932 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (2173 ms) 2023-10-30T12:14:09.932 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2023-10-30T12:14:09.933 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (2377 ms) 2023-10-30T12:14:09.933 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2023-10-30T12:14:09.933 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (2601 ms) 2023-10-30T12:14:09.933 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2023-10-30T12:14:09.933 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (21064 ms) 2023-10-30T12:14:09.934 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2023-10-30T12:14:09.934 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (8971 ms) 2023-10-30T12:14:09.934 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2023-10-30T12:14:09.934 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (2316 ms) 2023-10-30T12:14:09.934 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2023-10-30T12:14:09.935 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (4409 ms) 2023-10-30T12:14:09.935 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2023-10-30T12:14:09.935 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (1486 ms) 2023-10-30T12:14:09.935 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2023-10-30T12:14:09.935 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (2041 ms) 2023-10-30T12:14:09.936 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2023-10-30T12:14:09.936 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (1759 ms) 2023-10-30T12:14:09.936 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2023-10-30T12:14:09.936 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (578 ms) 2023-10-30T12:14:09.936 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-10-30T12:14:09.936 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (296 ms) 2023-10-30T12:14:09.937 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2023-10-30T12:14:09.937 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (1165 ms) 2023-10-30T12:14:09.937 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2023-10-30T12:14:09.937 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (51 ms) 2023-10-30T12:14:09.937 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2023-10-30T12:14:09.937 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (281 ms) 2023-10-30T12:14:09.938 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (72026 ms total) 2023-10-30T12:14:09.938 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: 2023-10-30T12:14:09.938 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2023-10-30T12:14:09.938 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2023-10-30T12:14:09.939 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (8097 ms) 2023-10-30T12:14:09.939 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2023-10-30T12:14:09.939 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (3137 ms) 2023-10-30T12:14:09.939 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2023-10-30T12:14:09.939 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (1271 ms) 2023-10-30T12:14:09.939 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2023-10-30T12:14:09.940 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (904 ms) 2023-10-30T12:14:09.940 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2023-10-30T12:14:10.131 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.R api_stat: Running main() from gmock_main.cc 2023-10-30T12:14:10.131 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2023-10-30T12:14:10.131 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [----------] Global test environment set-up. 2023-10-30T12:14:10.132 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [----------] 5 tests from LibRadosStat 2023-10-30T12:14:10.132 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2023-10-30T12:14:10.132 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ OK ] LibRadosStat.Stat (29930 ms) 2023-10-30T12:14:10.132 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2023-10-30T12:14:10.132 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (13922 ms) 2023-10-30T12:14:10.133 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2023-10-30T12:14:10.133 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ OK ] LibRadosStat.StatNS (22820 ms) 2023-10-30T12:14:10.133 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2023-10-30T12:14:10.133 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (7738 ms) 2023-10-30T12:14:10.133 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2023-10-30T12:14:10.134 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (2352 ms) 2023-10-30T12:14:10.134 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [----------] 5 tests from LibRadosStat (76762 ms total) 2023-10-30T12:14:10.134 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: 2023-10-30T12:14:10.134 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2023-10-30T12:14:10.135 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2023-10-30T12:14:10.135 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (6568 ms) 2023-10-30T12:14:10.135 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2023-10-30T12:14:10.135 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (5029 ms) 2023-10-30T12:14:10.135 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2023-10-30T12:14:10.136 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (3 ms) 2023-10-30T12:14:10.136 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2023-10-30T12:14:10.136 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (583 ms) 2023-10-30T12:14:10.136 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (12183 ms total) 2023-10-30T12:14:10.136 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: 2023-10-30T12:14:10.137 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [----------] Global test environment tear-down 2023-10-30T12:14:10.137 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (148748 ms total) 2023-10-30T12:14:10.137 INFO:tasks.workunit.client.0.smithi003.stdout: api_stat: [ PASSED ] 9 tests. 2023-10-30T12:14:18.685 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: Running main() from gmock_main.cc 2023-10-30T12:14:18.686 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2023-10-30T12:14:18.686 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [----------] Global test environment set-up. 2023-10-30T12:14:18.687 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [----------] 8 tests from LibRadosLock 2023-10-30T12:14:18.687 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2023-10-30T12:14:18.687 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (37214 ms) 2023-10-30T12:14:18.687 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2023-10-30T12:14:18.687 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLock.LockShared (8265 ms) 2023-10-30T12:14:18.688 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2023-10-30T12:14:18.688 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (11388 ms) 2023-10-30T12:14:18.688 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2023-10-30T12:14:18.688 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1835 ms) 2023-10-30T12:14:18.689 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2023-10-30T12:14:18.689 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (5135 ms) 2023-10-30T12:14:18.689 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2023-10-30T12:14:18.689 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLock.Unlock (2234 ms) 2023-10-30T12:14:18.689 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2023-10-30T12:14:18.690 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (2724 ms) 2023-10-30T12:14:18.690 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2023-10-30T12:14:18.690 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (865 ms) 2023-10-30T12:14:18.690 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [----------] 8 tests from LibRadosLock (69660 ms total) 2023-10-30T12:14:18.691 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: 2023-10-30T12:14:18.691 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2023-10-30T12:14:18.691 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2023-10-30T12:14:18.691 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (12845 ms) 2023-10-30T12:14:18.691 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2023-10-30T12:14:18.692 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (2696 ms) 2023-10-30T12:14:18.692 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2023-10-30T12:14:18.692 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (3377 ms) 2023-10-30T12:14:18.692 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2023-10-30T12:14:18.693 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1605 ms) 2023-10-30T12:14:18.693 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2023-10-30T12:14:18.693 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (3148 ms) 2023-10-30T12:14:18.693 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2023-10-30T12:14:18.693 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (1868 ms) 2023-10-30T12:14:18.694 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2023-10-30T12:14:18.694 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (1473 ms) 2023-10-30T12:14:18.694 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2023-10-30T12:14:18.694 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (907 ms) 2023-10-30T12:14:18.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (27919 ms total) 2023-10-30T12:14:18.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: 2023-10-30T12:14:18.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [----------] Global test environment tear-down 2023-10-30T12:14:18.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (157440 ms total) 2023-10-30T12:14:18.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_lock: [ PASSED ] 16 tests. 2023-10-30T12:14:24.433 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: Running main() from gmock_main.cc 2023-10-30T12:14:24.434 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2023-10-30T12:14:24.434 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [----------] Global test environment set-up. 2023-10-30T12:14:24.434 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [----------] 14 tests from LibRadosIo 2023-10-30T12:14:24.435 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2023-10-30T12:14:24.435 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (14013 ms) 2023-10-30T12:14:24.435 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2023-10-30T12:14:24.435 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2023-10-30T12:14:24.435 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2023-10-30T12:14:24.436 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: no timeout :/ 2023-10-30T12:14:24.436 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: no timeout :/ 2023-10-30T12:14:24.436 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: no timeout :/ 2023-10-30T12:14:24.436 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: no timeout :/ 2023-10-30T12:14:24.436 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: no timeout :/ 2023-10-30T12:14:24.437 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (1594 ms) 2023-10-30T12:14:24.437 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2023-10-30T12:14:24.437 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (5910 ms) 2023-10-30T12:14:24.437 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2023-10-30T12:14:24.437 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIo.Checksum (3893 ms) 2023-10-30T12:14:24.438 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2023-10-30T12:14:24.438 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (7244 ms) 2023-10-30T12:14:24.438 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2023-10-30T12:14:24.438 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (22966 ms) 2023-10-30T12:14:24.439 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2023-10-30T12:14:24.439 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (2935 ms) 2023-10-30T12:14:24.439 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2023-10-30T12:14:24.439 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (10060 ms) 2023-10-30T12:14:24.439 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2023-10-30T12:14:24.439 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIo.TruncTest (372 ms) 2023-10-30T12:14:24.440 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2023-10-30T12:14:24.440 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (74 ms) 2023-10-30T12:14:24.440 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2023-10-30T12:14:24.440 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (1462 ms) 2023-10-30T12:14:24.440 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2023-10-30T12:14:24.441 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIo.RmXattr (284 ms) 2023-10-30T12:14:24.441 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2023-10-30T12:14:24.441 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIo.XattrIter (193 ms) 2023-10-30T12:14:24.441 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [----------] 14 tests from LibRadosIo (71001 ms total) 2023-10-30T12:14:24.441 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: 2023-10-30T12:14:24.442 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2023-10-30T12:14:24.442 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2023-10-30T12:14:24.442 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (9221 ms) 2023-10-30T12:14:24.442 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2023-10-30T12:14:24.442 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (4196 ms) 2023-10-30T12:14:24.443 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2023-10-30T12:14:24.443 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (3720 ms) 2023-10-30T12:14:24.443 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2023-10-30T12:14:24.443 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (1764 ms) 2023-10-30T12:14:24.443 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2023-10-30T12:14:24.444 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (471 ms) 2023-10-30T12:14:24.444 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2023-10-30T12:14:24.444 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (1723 ms) 2023-10-30T12:14:24.444 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2023-10-30T12:14:24.444 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (214 ms) 2023-10-30T12:14:24.445 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2023-10-30T12:14:24.445 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (1856 ms) 2023-10-30T12:14:24.445 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2023-10-30T12:14:24.445 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (2153 ms) 2023-10-30T12:14:24.445 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2023-10-30T12:14:24.446 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (1372 ms) 2023-10-30T12:14:24.446 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [----------] 10 tests from LibRadosIoEC (26690 ms total) 2023-10-30T12:14:24.446 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: 2023-10-30T12:14:24.447 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [----------] Global test environment tear-down 2023-10-30T12:14:24.447 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [==========] 24 tests from 2 test suites ran. (163224 ms total) 2023-10-30T12:14:24.447 INFO:tasks.workunit.client.0.smithi003.stdout: api_io: [ PASSED ] 24 tests. 2023-10-30T12:14:25.477 INFO:tasks.workunit.client.0.smithi003.stdout:oundTripPP2 (2338 ms) 2023-10-30T12:14:25.477 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2023-10-30T12:14:25.478 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (2334 ms) 2023-10-30T12:14:25.478 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2023-10-30T12:14:25.478 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (2112 ms) 2023-10-30T12:14:25.478 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2023-10-30T12:14:25.478 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (779 ms) 2023-10-30T12:14:25.479 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2023-10-30T12:14:25.479 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (186 ms) 2023-10-30T12:14:25.479 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2023-10-30T12:14:25.479 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (2957 ms) 2023-10-30T12:14:25.479 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2023-10-30T12:14:25.480 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (1059 ms) 2023-10-30T12:14:25.480 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2023-10-30T12:14:25.480 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (166 ms) 2023-10-30T12:14:25.480 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2023-10-30T12:14:25.480 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (1319 ms) 2023-10-30T12:14:25.481 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2023-10-30T12:14:25.481 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (689 ms) 2023-10-30T12:14:25.481 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2023-10-30T12:14:25.481 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (164 ms) 2023-10-30T12:14:25.482 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2023-10-30T12:14:25.482 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (95 ms) 2023-10-30T12:14:25.482 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2023-10-30T12:14:25.482 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (123 ms) 2023-10-30T12:14:25.482 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (27730 ms total) 2023-10-30T12:14:25.483 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: 2023-10-30T12:14:25.483 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [----------] Global test environment tear-down 2023-10-30T12:14:25.483 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (164263 ms total) 2023-10-30T12:14:25.483 INFO:tasks.workunit.client.0.smithi003.stdout: api_io_pp: [ PASSED ] 37 tests. 2023-10-30T12:14:32.547 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:05:32.482 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:32.552 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:05:32.487 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:32.611 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:05:32.547 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:32.613 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:05:32.548 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:32.624 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:05:32.533 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:32.629 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:05:32.538 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:32.719 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:05:32.654 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:32.764 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:05:32.673 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:32.771 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:05:32.706 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:49.665 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: Running main() from gmock_main.cc 2023-10-30T12:14:49.666 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2023-10-30T12:14:49.666 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [----------] Global test environment set-up. 2023-10-30T12:14:49.666 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2023-10-30T12:14:49.667 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2023-10-30T12:14:49.667 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (20572 ms) 2023-10-30T12:14:49.667 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2023-10-30T12:14:49.667 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (9753 ms) 2023-10-30T12:14:49.667 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2023-10-30T12:14:49.668 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (10920 ms) 2023-10-30T12:14:49.668 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2023-10-30T12:14:49.668 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (6403 ms) 2023-10-30T12:14:49.668 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (47648 ms total) 2023-10-30T12:14:49.668 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: 2023-10-30T12:14:49.669 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2023-10-30T12:14:49.669 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2023-10-30T12:14:49.669 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (58500 ms) 2023-10-30T12:14:49.669 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2023-10-30T12:14:49.669 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (30033 ms) 2023-10-30T12:14:49.669 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2023-10-30T12:14:49.670 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (8864 ms) 2023-10-30T12:14:49.670 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (97397 ms total) 2023-10-30T12:14:49.670 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: 2023-10-30T12:14:49.670 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2023-10-30T12:14:49.670 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2023-10-30T12:14:49.671 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (4368 ms) 2023-10-30T12:14:49.671 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2023-10-30T12:14:49.671 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2387 ms) 2023-10-30T12:14:49.671 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2023-10-30T12:14:49.671 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (3015 ms) 2023-10-30T12:14:49.672 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2023-10-30T12:14:49.672 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2080 ms) 2023-10-30T12:14:49.672 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (11850 ms total) 2023-10-30T12:14:49.672 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: 2023-10-30T12:14:49.673 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2023-10-30T12:14:49.673 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2023-10-30T12:14:49.673 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (5356 ms) 2023-10-30T12:14:49.673 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2023-10-30T12:14:49.673 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (6676 ms) 2023-10-30T12:14:49.674 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (12032 ms total) 2023-10-30T12:14:49.674 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: 2023-10-30T12:14:49.674 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [----------] Global test environment tear-down 2023-10-30T12:14:49.674 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (188318 ms total) 2023-10-30T12:14:49.675 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots: [ PASSED ] 13 tests. 2023-10-30T12:14:52.700 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:05:52.636 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:52.772 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:05:52.708 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:52.852 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:05:52.760 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:52.854 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:05:52.790 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:52.855 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:05:52.763 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:52.858 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:05:52.793 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:52.922 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:05:52.858 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:52.930 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:05:52.839 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:14:52.936 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:05:52.871 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:11.278 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2023-10-30T12:15:11.279 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [==========] Running 19 tests from 4 test suites. 2023-10-30T12:15:11.279 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2023-10-30T12:15:11.279 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2023-10-30T12:15:11.280 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: seed 92228 2023-10-30T12:15:11.280 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2023-10-30T12:15:11.280 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (20096 ms) 2023-10-30T12:15:11.280 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2023-10-30T12:15:11.280 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (8906 ms) 2023-10-30T12:15:11.281 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2023-10-30T12:15:11.281 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (11714 ms) 2023-10-30T12:15:11.281 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2023-10-30T12:15:11.281 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (5569 ms) 2023-10-30T12:15:11.281 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2023-10-30T12:15:11.282 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (9602 ms) 2023-10-30T12:15:11.282 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (55887 ms total) 2023-10-30T12:15:11.282 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: 2023-10-30T12:15:11.282 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2023-10-30T12:15:11.282 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2023-10-30T12:15:11.283 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (58748 ms) 2023-10-30T12:15:11.283 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2023-10-30T12:15:11.283 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (23215 ms) 2023-10-30T12:15:11.283 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2023-10-30T12:15:11.283 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (11934 ms) 2023-10-30T12:15:11.284 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2023-10-30T12:15:11.284 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4544 ms) 2023-10-30T12:15:11.284 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2023-10-30T12:15:11.284 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (4080 ms) 2023-10-30T12:15:11.284 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-10-30T12:15:11.285 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6839-gcc36dd85/rpm/el9/BUILD/ceph-18.0.0-6839-gcc36dd85/src/test/librados/snapshots_cxx.cc:463: Skipped 2023-10-30T12:15:11.285 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: 2023-10-30T12:15:11.285 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2023-10-30T12:15:11.285 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2023-10-30T12:15:11.285 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi003-92228-7 2023-10-30T12:15:11.286 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: waiting for snaps to purge 2023-10-30T12:15:11.286 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (19415 ms) 2023-10-30T12:15:11.286 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (121936 ms total) 2023-10-30T12:15:11.286 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: 2023-10-30T12:15:11.287 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2023-10-30T12:15:11.287 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2023-10-30T12:15:11.287 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (5023 ms) 2023-10-30T12:15:11.287 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2023-10-30T12:15:11.288 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (3214 ms) 2023-10-30T12:15:11.288 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2023-10-30T12:15:11.288 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (3807 ms) 2023-10-30T12:15:11.288 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2023-10-30T12:15:11.288 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2137 ms) 2023-10-30T12:15:11.289 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (14181 ms total) 2023-10-30T12:15:11.289 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: 2023-10-30T12:15:11.289 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2023-10-30T12:15:12.868 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:12.803 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:12.870 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:12.805 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:12.932 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:12.841 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:12.933 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:12.842 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:12.949 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:12.884 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:12.954 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:12.890 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:12.965 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:12.900 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:12.996 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:12.904 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:13.033 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:12.967 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:23.950 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN api_aio: Running main() from gmock_main.cc 2023-10-30T12:15:23.951 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2023-10-30T12:15:23.951 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [----------] Global test environment set-up. 2023-10-30T12:15:23.952 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [----------] 26 tests from LibRadosAio 2023-10-30T12:15:23.952 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2023-10-30T12:15:23.952 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.TooBig (4076 ms) 2023-10-30T12:15:23.952 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2023-10-30T12:15:23.952 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (47758 ms) 2023-10-30T12:15:23.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2023-10-30T12:15:23.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (6373 ms) 2023-10-30T12:15:23.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2023-10-30T12:15:23.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (23082 ms) 2023-10-30T12:15:23.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2023-10-30T12:15:23.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (47928 ms) 2023-10-30T12:15:23.954 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2023-10-30T12:15:23.954 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (11324 ms) 2023-10-30T12:15:23.954 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2023-10-30T12:15:23.954 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3800 ms) 2023-10-30T12:15:23.954 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2023-10-30T12:15:23.955 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (4583 ms) 2023-10-30T12:15:23.955 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2023-10-30T12:15:23.955 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (5387 ms) 2023-10-30T12:15:23.955 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2023-10-30T12:15:23.955 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (4138 ms) 2023-10-30T12:15:23.956 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2023-10-30T12:15:23.956 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (4656 ms) 2023-10-30T12:15:23.956 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2023-10-30T12:15:23.956 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (4076 ms) 2023-10-30T12:15:23.956 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2023-10-30T12:15:23.957 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (5307 ms) 2023-10-30T12:15:23.957 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2023-10-30T12:15:23.957 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3109 ms) 2023-10-30T12:15:23.957 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2023-10-30T12:15:23.958 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.Flush (3510 ms) 2023-10-30T12:15:23.958 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2023-10-30T12:15:23.958 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3445 ms) 2023-10-30T12:15:23.958 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2023-10-30T12:15:23.959 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3801 ms) 2023-10-30T12:15:23.959 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2023-10-30T12:15:23.959 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (4279 ms) 2023-10-30T12:15:23.959 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2023-10-30T12:15:23.959 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (4023 ms) 2023-10-30T12:15:23.960 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2023-10-30T12:15:23.960 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (5284 ms) 2023-10-30T12:15:23.960 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2023-10-30T12:15:23.960 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3881 ms) 2023-10-30T12:15:23.960 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2023-10-30T12:15:23.961 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3107 ms) 2023-10-30T12:15:23.961 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2023-10-30T12:15:23.961 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (4250 ms) 2023-10-30T12:15:23.961 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2023-10-30T12:15:23.961 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3761 ms) 2023-10-30T12:15:23.961 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2023-10-30T12:15:23.962 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (4136 ms) 2023-10-30T12:15:23.962 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2023-10-30T12:15:23.962 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3677 ms) 2023-10-30T12:15:23.962 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [----------] 26 tests from LibRadosAio (222752 ms total) 2023-10-30T12:15:23.962 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: 2023-10-30T12:15:23.963 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2023-10-30T12:15:29.248 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWr ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2023-10-30T12:15:29.248 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (5355 ms) 2023-10-30T12:15:29.249 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2023-10-30T12:15:29.249 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4609 ms) 2023-10-30T12:15:29.249 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2023-10-30T12:15:29.249 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4760 ms) 2023-10-30T12:15:29.249 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (14724 ms total) 2023-10-30T12:15:29.250 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: 2023-10-30T12:15:29.250 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2023-10-30T12:15:29.250 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [==========] 19 tests from 4 test suites ran. (227894 ms total) 2023-10-30T12:15:29.250 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ PASSED ] 18 tests. 2023-10-30T12:15:29.250 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2023-10-30T12:15:29.251 INFO:tasks.workunit.client.0.smithi003.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-10-30T12:15:32.472 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: Running main() from gmock_main.cc 2023-10-30T12:15:32.472 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2023-10-30T12:15:32.472 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [----------] Global test environment set-up. 2023-10-30T12:15:32.472 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2023-10-30T12:15:32.473 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2023-10-30T12:15:32.473 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (4091 ms) 2023-10-30T12:15:32.473 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2023-10-30T12:15:32.473 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (48853 ms) 2023-10-30T12:15:32.473 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2023-10-30T12:15:32.474 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (40789 ms) 2023-10-30T12:15:32.474 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2023-10-30T12:15:32.474 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (39733 ms) 2023-10-30T12:15:32.474 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2023-10-30T12:15:32.474 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (7149 ms) 2023-10-30T12:15:32.475 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2023-10-30T12:15:32.475 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3720 ms) 2023-10-30T12:15:32.475 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2023-10-30T12:15:32.475 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (4575 ms) 2023-10-30T12:15:32.475 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2023-10-30T12:15:32.476 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3298 ms) 2023-10-30T12:15:32.476 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2023-10-30T12:15:32.476 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3907 ms) 2023-10-30T12:15:32.476 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2023-10-30T12:15:32.476 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (4558 ms) 2023-10-30T12:15:32.476 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2023-10-30T12:15:32.477 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (4509 ms) 2023-10-30T12:15:32.477 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2023-10-30T12:15:32.477 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (4183 ms) 2023-10-30T12:15:32.477 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2023-10-30T12:15:32.477 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (4196 ms) 2023-10-30T12:15:32.477 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2023-10-30T12:15:32.478 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3308 ms) 2023-10-30T12:15:32.478 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2023-10-30T12:15:32.478 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (4320 ms) 2023-10-30T12:15:32.478 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2023-10-30T12:15:32.478 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3408 ms) 2023-10-30T12:15:32.479 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2023-10-30T12:15:32.479 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3874 ms) 2023-10-30T12:15:32.479 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2023-10-30T12:15:32.479 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (4186 ms) 2023-10-30T12:15:32.479 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2023-10-30T12:15:32.480 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (5217 ms) 2023-10-30T12:15:32.480 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2023-10-30T12:15:32.480 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (4791 ms) 2023-10-30T12:15:32.480 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2023-10-30T12:15:32.480 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (4250 ms) 2023-10-30T12:15:32.480 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2023-10-30T12:15:32.481 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (5767 ms) 2023-10-30T12:15:32.481 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2023-10-30T12:15:32.481 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (4341 ms) 2023-10-30T12:15:32.481 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2023-10-30T12:15:32.481 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (4060 ms) 2023-10-30T12:15:32.481 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2023-10-30T12:15:32.482 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2670 ms) 2023-10-30T12:15:32.482 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2023-10-30T12:15:32.482 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (4286 ms) 2023-10-30T12:15:32.482 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2023-10-30T12:15:33.054 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:32.989 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:33.057 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:32.992 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:33.113 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:33.049 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:33.123 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:33.059 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:33.127 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:33.036 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:33.129 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:33.064 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:33.136 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:33.045 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:33.192 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:33.100 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:33.196 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:33.132 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:53.059 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:52.995 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:53.063 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:52.998 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:53.112 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:53.020 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:53.115 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:53.024 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:53.129 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:53.065 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:53.131 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:53.067 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:53.135 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:53.071 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:53.166 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:53.074 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:15:53.191 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:53.126 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:08.692 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosA api_list: : cursor()=4:b29083e3:::5:head expected=4:b29083e3:::5:head 2023-10-30T12:16:08.692 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:b29083e3:::5:head -> 5 2023-10-30T12:16:08.692 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=5 expected=5 2023-10-30T12:16:08.693 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:89d3ae78:::11:head 2023-10-30T12:16:08.693 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : cursor()=4:89d3ae78:::11:head expected=4:89d3ae78:::11:head 2023-10-30T12:16:08.693 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:89d3ae78:::11:head -> 11 2023-10-30T12:16:08.693 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=11 expected=11 2023-10-30T12:16:08.693 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:863748b0:::15:head 2023-10-30T12:16:08.694 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : cursor()=4:863748b0:::15:head expected=4:863748b0:::15:head 2023-10-30T12:16:08.694 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:863748b0:::15:head -> 15 2023-10-30T12:16:08.694 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=15 expected=15 2023-10-30T12:16:08.694 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:5c6b0b28:::7:head 2023-10-30T12:16:08.694 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : cursor()=4:5c6b0b28:::7:head expected=4:5c6b0b28:::7:head 2023-10-30T12:16:08.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:5c6b0b28:::7:head -> 7 2023-10-30T12:16:08.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=7 expected=7 2023-10-30T12:16:08.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:f905c69b:::2:head 2023-10-30T12:16:08.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : cursor()=4:f905c69b:::2:head expected=4:f905c69b:::2:head 2023-10-30T12:16:08.695 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:f905c69b:::2:head -> 2 2023-10-30T12:16:08.696 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=2 expected=2 2023-10-30T12:16:08.696 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:e960b815:::9:head 2023-10-30T12:16:08.696 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : cursor()=4:e960b815:::9:head expected=4:e960b815:::9:head 2023-10-30T12:16:08.696 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:e960b815:::9:head -> 9 2023-10-30T12:16:08.696 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=9 expected=9 2023-10-30T12:16:08.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:de5d7c5f:::12:head 2023-10-30T12:16:08.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : cursor()=4:de5d7c5f:::12:head expected=4:de5d7c5f:::12:head 2023-10-30T12:16:08.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:de5d7c5f:::12:head -> 12 2023-10-30T12:16:08.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=12 expected=12 2023-10-30T12:16:08.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : seek to 4:d83876eb:::4:head 2023-10-30T12:16:08.697 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : cursor()=4:d83876eb:::4:head expected=4:d83876eb:::4:head 2023-10-30T12:16:08.698 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: > 4:d83876eb:::4:head -> 4 2023-10-30T12:16:08.698 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: : entry=4 expected=4 2023-10-30T12:16:08.698 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (17247 ms) 2023-10-30T12:16:08.698 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2023-10-30T12:16:08.699 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (135714 ms) 2023-10-30T12:16:08.699 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2023-10-30T12:16:08.699 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: split 0/5 -> MIN 4:33333333::::head 2023-10-30T12:16:08.699 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: split 1/5 -> 4:33333333::::head 4:66666666::::head 2023-10-30T12:16:08.699 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: split 2/5 -> 4:66666666::::head 4:99999999::::head 2023-10-30T12:16:08.699 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: split 3/5 -> 4:99999999::::head 4:cccccccc::::head 2023-10-30T12:16:08.700 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: split 4/5 -> 4:cccccccc::::head MAX 2023-10-30T12:16:08.700 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (22239 ms) 2023-10-30T12:16:08.700 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [----------] 7 tests from LibRadosList (252657 ms total) 2023-10-30T12:16:08.700 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 2023-10-30T12:16:08.700 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [----------] 3 tests from LibRadosListEC 2023-10-30T12:16:08.701 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2023-10-30T12:16:08.701 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1971 ms) 2023-10-30T12:16:08.701 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2023-10-30T12:16:08.701 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: myset foo1,foo2,foo3 2023-10-30T12:16:08.701 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo1 2023-10-30T12:16:08.701 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo2 2023-10-30T12:16:08.702 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo3 2023-10-30T12:16:08.702 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: myset foo1,foo4,foo5 2023-10-30T12:16:08.702 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo4 2023-10-30T12:16:08.702 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo5 2023-10-30T12:16:08.702 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo1 2023-10-30T12:16:08.703 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: myset foo6,foo7 2023-10-30T12:16:08.703 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo7 2023-10-30T12:16:08.703 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: foo6 2023-10-30T12:16:08.703 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-10-30T12:16:08.703 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: ns1:foo4 2023-10-30T12:16:08.703 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: ns1:foo5 2023-10-30T12:16:08.704 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: ns2:foo7 2023-10-30T12:16:08.704 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: ns2:foo6 2023-10-30T12:16:08.704 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: ns1:foo1 2023-10-30T12:16:08.704 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: :foo1 2023-10-30T12:16:08.704 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: :foo2 2023-10-30T12:16:08.705 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: :foo3 2023-10-30T12:16:08.705 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (2962 ms) 2023-10-30T12:16:08.705 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2023-10-30T12:16:08.705 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 1 0 2023-10-30T12:16:08.705 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 10 0 2023-10-30T12:16:08.706 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 13 0 2023-10-30T12:16:13.211 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:13.146 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:13.214 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:13.150 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:13.266 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:13.174 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:13.269 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:13.204 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:13.269 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:13.178 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:13.272 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:13.208 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:13.293 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:13.227 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:13.316 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:13.225 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:13.334 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:13.270 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:16.876 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 7 0 2023-10-30T12:16:16.876 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 14 0 2023-10-30T12:16:16.876 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 0 0 2023-10-30T12:16:16.877 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 15 0 2023-10-30T12:16:16.877 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 11 0 2023-10-30T12:16:16.877 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 5 0 2023-10-30T12:16:16.877 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 8 0 2023-10-30T12:16:16.877 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 6 0 2023-10-30T12:16:16.878 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 3 0 2023-10-30T12:16:16.878 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 4 0 2023-10-30T12:16:16.878 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 12 0 2023-10-30T12:16:16.878 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 9 0 2023-10-30T12:16:16.878 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 2 0 2023-10-30T12:16:16.879 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-10-30T12:16:16.879 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (3718 ms) 2023-10-30T12:16:16.879 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [----------] 3 tests from LibRadosListEC (8651 ms total) 2023-10-30T12:16:16.879 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 2023-10-30T12:16:16.879 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [----------] 1 test from LibRadosListNP 2023-10-30T12:16:16.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2023-10-30T12:16:16.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (4298 ms) 2023-10-30T12:16:16.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [----------] 1 test from LibRadosListNP (4298 ms total) 2023-10-30T12:16:16.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: 2023-10-30T12:16:16.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [----------] Global test environment tear-down 2023-10-30T12:16:16.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [==========] 11 tests from 3 test suites ran. (275629 ms total) 2023-10-30T12:16:16.881 INFO:tasks.workunit.client.0.smithi003.stdout: api_list: [ PASSED ] 11 tests. 2023-10-30T12:16:24.935 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: Running main() from gmock_main.cc 2023-10-30T12:16:24.936 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2023-10-30T12:16:24.936 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [----------] Global test environment set-up. 2023-10-30T12:16:24.936 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2023-10-30T12:16:24.936 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2023-10-30T12:16:24.936 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2023-10-30T12:16:24.937 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2023-10-30T12:16:24.937 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: 2023-10-30T12:16:24.937 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2023-10-30T12:16:24.937 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2023-10-30T12:16:24.937 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (23264 ms) 2023-10-30T12:16:24.937 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2023-10-30T12:16:24.938 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (15748 ms) 2023-10-30T12:16:24.938 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2023-10-30T12:16:24.938 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (24328 ms) 2023-10-30T12:16:24.938 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2023-10-30T12:16:24.938 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (44607 ms) 2023-10-30T12:16:24.939 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2023-10-30T12:16:24.939 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (30023 ms) 2023-10-30T12:16:24.939 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2023-10-30T12:16:24.939 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (8295 ms) 2023-10-30T12:16:24.939 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2023-10-30T12:16:24.940 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (137177 ms) 2023-10-30T12:16:24.940 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (283442 ms total) 2023-10-30T12:16:24.940 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: 2023-10-30T12:16:24.940 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [----------] Global test environment tear-down 2023-10-30T12:16:24.940 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (283443 ms total) 2023-10-30T12:16:24.940 INFO:tasks.workunit.client.0.smithi003.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2023-10-30T12:16:26.074 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2023-10-30T12:16:26.074 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] Global test environment set-up. 2023-10-30T12:16:26.075 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2023-10-30T12:16:26.075 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2023-10-30T12:16:26.075 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2023-10-30T12:16:26.075 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-10-30T12:16:26.075 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: 2023-10-30T12:16:26.076 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2023-10-30T12:16:26.076 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: seed 92129 2023-10-30T12:16:26.076 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2023-10-30T12:16:26.076 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (308 ms) 2023-10-30T12:16:26.076 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2023-10-30T12:16:26.077 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (19504 ms) 2023-10-30T12:16:26.077 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2023-10-30T12:16:26.077 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (396 ms) 2023-10-30T12:16:26.077 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2023-10-30T12:16:26.077 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (21515 ms) 2023-10-30T12:16:26.078 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2023-10-30T12:16:26.078 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (12904 ms) 2023-10-30T12:16:26.078 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2023-10-30T12:16:26.078 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (1409 ms) 2023-10-30T12:16:26.078 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2023-10-30T12:16:26.079 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (1027 ms) 2023-10-30T12:16:26.079 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2023-10-30T12:16:26.079 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (2727 ms) 2023-10-30T12:16:26.079 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2023-10-30T12:16:26.080 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (1899 ms) 2023-10-30T12:16:26.080 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2023-10-30T12:16:26.080 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (5132 ms) 2023-10-30T12:16:26.080 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2023-10-30T12:16:26.080 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (48 ms) 2023-10-30T12:16:26.081 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2023-10-30T12:16:26.081 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (4656 ms) 2023-10-30T12:16:26.081 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2023-10-30T12:16:26.081 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (248 ms) 2023-10-30T12:16:26.081 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2023-10-30T12:16:26.082 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: osd_max_attr_size = 0 2023-10-30T12:16:26.082 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2023-10-30T12:16:26.082 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (106169 ms) 2023-10-30T12:16:26.082 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2023-10-30T12:16:26.082 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (13592 ms) 2023-10-30T12:16:26.083 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2023-10-30T12:16:26.083 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: waiting for initial deep scrubs... 2023-10-30T12:16:26.083 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: done waiting, doing copies 2023-10-30T12:16:26.083 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: waiting for final deep scrubs... 2023-10-30T12:16:26.083 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: done waiting 2023-10-30T12:16:26.084 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (83094 ms) 2023-10-30T12:16:26.084 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2023-10-30T12:16:26.084 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (419 ms) 2023-10-30T12:16:26.084 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2023-10-30T12:16:26.084 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (42 ms) 2023-10-30T12:16:26.085 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2023-10-30T12:16:26.085 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5680 ms) 2023-10-30T12:16:26.085 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2023-10-30T12:16:26.085 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2023-10-30T12:16:26.085 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2023-10-30T12:16:26.086 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2023-10-30T12:16:26.086 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2023-10-30T12:16:26.086 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2023-10-30T12:16:33.387 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:33.322 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:33.389 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:33.325 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:33.435 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:33.370 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:33.441 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:33.376 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:33.443 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:33.378 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:33.454 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:33.363 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:33.456 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:33.364 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:33.494 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:33.430 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:33.513 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:33.422 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:38.699 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:38.635 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:38.701 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:38.637 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:38.729 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:38.665 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:38.764 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:38.673 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:38.767 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:38.703 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:38.768 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:38.704 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:38.776 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:38.685 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:38.794 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:38.730 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:38.796 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:38.705 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:53.475 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:53.410 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:53.476 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:53.412 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:53.512 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:53.448 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:53.519 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:53.428 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:53.522 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:53.431 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:53.524 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:53.459 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:53.525 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:53.461 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:53.558 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:53.467 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:53.562 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:53.498 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:16:55.948 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP (280770 ms total) 2023-10-30T12:16:55.948 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: 2023-10-30T12:16:55.949 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2023-10-30T12:16:55.949 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2023-10-30T12:16:55.949 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (2408 ms) 2023-10-30T12:16:55.949 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (2408 ms total) 2023-10-30T12:16:55.950 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: 2023-10-30T12:16:55.950 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2023-10-30T12:16:55.950 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2023-10-30T12:16:55.950 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (518 ms) 2023-10-30T12:16:55.950 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2023-10-30T12:16:55.951 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (51 ms) 2023-10-30T12:16:55.951 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (569 ms total) 2023-10-30T12:16:55.951 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: 2023-10-30T12:16:55.951 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2023-10-30T12:16:55.952 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2023-10-30T12:16:55.952 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (659 ms) 2023-10-30T12:16:55.952 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2023-10-30T12:16:55.952 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (39 ms) 2023-10-30T12:16:55.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (698 ms total) 2023-10-30T12:16:55.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: 2023-10-30T12:16:55.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2023-10-30T12:16:55.953 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2023-10-30T12:16:55.954 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (456 ms) 2023-10-30T12:16:55.954 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2023-10-30T12:16:55.954 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (218 ms) 2023-10-30T12:16:55.954 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (674 ms total) 2023-10-30T12:16:55.955 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: 2023-10-30T12:16:55.955 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2023-10-30T12:16:55.955 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2023-10-30T12:16:55.955 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1498 ms) 2023-10-30T12:16:55.955 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1498 ms total) 2023-10-30T12:16:55.956 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: 2023-10-30T12:16:55.956 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [----------] Global test environment tear-down 2023-10-30T12:16:55.956 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (314616 ms total) 2023-10-30T12:16:55.957 INFO:tasks.workunit.client.0.smithi003.stdout: api_misc_pp: [ PASSED ] 31 tests. 2023-10-30T12:17:13.543 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:13.478 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:13.546 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:13.481 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:13.548 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:13.484 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:13.550 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:13.486 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:13.593 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:08:13.502 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:13.594 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:08:13.503 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:13.611 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:13.547 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:13.613 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:13.549 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:13.632 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:08:13.541 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:24.769 INFO:tasks.workunit.client.0.smithi003.stdout:ite 2023-10-30T12:17:24.769 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7469 ms) 2023-10-30T12:17:24.769 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2023-10-30T12:17:24.770 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7099 ms) 2023-10-30T12:17:24.770 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2023-10-30T12:17:24.770 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (8065 ms) 2023-10-30T12:17:24.770 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2023-10-30T12:17:24.771 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (8135 ms) 2023-10-30T12:17:24.771 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2023-10-30T12:17:24.771 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7801 ms) 2023-10-30T12:17:24.771 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2023-10-30T12:17:24.771 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (6845 ms) 2023-10-30T12:17:24.772 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2023-10-30T12:17:24.772 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (8584 ms) 2023-10-30T12:17:24.772 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2023-10-30T12:17:24.772 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6993 ms) 2023-10-30T12:17:24.772 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2023-10-30T12:17:24.773 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7992 ms) 2023-10-30T12:17:24.773 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2023-10-30T12:17:24.773 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7792 ms) 2023-10-30T12:17:24.773 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2023-10-30T12:17:24.773 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7498 ms) 2023-10-30T12:17:24.774 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2023-10-30T12:17:24.774 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7723 ms) 2023-10-30T12:17:24.774 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2023-10-30T12:17:24.774 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6796 ms) 2023-10-30T12:17:24.774 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2023-10-30T12:17:24.775 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6842 ms) 2023-10-30T12:17:24.775 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2023-10-30T12:17:24.775 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7936 ms) 2023-10-30T12:17:24.775 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2023-10-30T12:17:24.775 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7246 ms) 2023-10-30T12:17:24.776 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (120816 ms total) 2023-10-30T12:17:24.776 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: 2023-10-30T12:17:24.776 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [----------] Global test environment tear-down 2023-10-30T12:17:24.776 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (343568 ms total) 2023-10-30T12:17:24.776 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio: [ PASSED ] 42 tests. 2023-10-30T12:17:33.820 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:33.755 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:33.824 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:33.759 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:33.868 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:08:33.777 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:33.870 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:08:33.779 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:33.882 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:33.817 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:33.891 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:33.827 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:33.893 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:33.829 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:33.912 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:08:33.821 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:33.931 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:33.866 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:53.937 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:53.872 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:53.947 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:53.882 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:53.951 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:53.887 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:53.953 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:08:53.862 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:53.957 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:53.892 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:53.957 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:08:53.866 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:54.041 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:08:53.950 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:54.071 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:54.006 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:17:54.080 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:54.016 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:12.714 INFO:tasks.workunit.client.0.smithi003.stdout:io.RoundTripAppendPP (3229 ms) 2023-10-30T12:18:12.715 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2023-10-30T12:18:12.715 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2600 ms) 2023-10-30T12:18:12.715 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2023-10-30T12:18:12.715 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3331 ms) 2023-10-30T12:18:12.715 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2023-10-30T12:18:12.716 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (4342 ms) 2023-10-30T12:18:12.716 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2023-10-30T12:18:12.716 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: setting pool EIO 2023-10-30T12:18:12.716 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: max_success 0, min_failed 1 2023-10-30T12:18:12.717 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4949 ms) 2023-10-30T12:18:12.717 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2023-10-30T12:18:12.717 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4445 ms) 2023-10-30T12:18:12.717 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (250935 ms total) 2023-10-30T12:18:12.717 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: 2023-10-30T12:18:12.717 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2023-10-30T12:18:12.718 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2023-10-30T12:18:12.718 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3557 ms) 2023-10-30T12:18:12.718 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2023-10-30T12:18:12.718 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9850 ms) 2023-10-30T12:18:12.719 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2023-10-30T12:18:12.719 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (17150 ms) 2023-10-30T12:18:12.719 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2023-10-30T12:18:12.719 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (4325 ms) 2023-10-30T12:18:12.720 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (34882 ms total) 2023-10-30T12:18:12.720 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: 2023-10-30T12:18:12.720 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2023-10-30T12:18:12.720 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-10-30T12:18:12.720 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3750 ms) 2023-10-30T12:18:12.721 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3750 ms total) 2023-10-30T12:18:12.721 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: 2023-10-30T12:18:12.721 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2023-10-30T12:18:12.721 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2023-10-30T12:18:12.721 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (15496 ms) 2023-10-30T12:18:12.722 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2023-10-30T12:18:12.722 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6355 ms) 2023-10-30T12:18:12.722 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2023-10-30T12:18:12.722 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6991 ms) 2023-10-30T12:18:12.722 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2023-10-30T12:18:12.723 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7293 ms) 2023-10-30T12:18:12.723 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2023-10-30T12:18:12.723 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (2745 ms) 2023-10-30T12:18:12.723 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2023-10-30T12:18:12.723 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7885 ms) 2023-10-30T12:18:12.724 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2023-10-30T12:18:12.724 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7351 ms) 2023-10-30T12:18:12.725 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2023-10-30T12:18:12.726 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7202 ms) 2023-10-30T12:18:12.726 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2023-10-30T12:18:12.726 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6976 ms) 2023-10-30T12:18:12.726 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2023-10-30T12:18:12.727 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (8299 ms) 2023-10-30T12:18:12.727 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2023-10-30T12:18:12.727 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7533 ms) 2023-10-30T12:18:12.727 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2023-10-30T12:18:12.727 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7670 ms) 2023-10-30T12:18:12.728 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2023-10-30T12:18:12.728 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6852 ms) 2023-10-30T12:18:12.728 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2023-10-30T12:18:14.061 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:13.997 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:14.110 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:14.046 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:14.112 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:14.048 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:14.113 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:14.048 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:14.153 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:14.088 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:14.153 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:14.089 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:14.160 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:14.069 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:14.161 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:14.070 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:14.206 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:14.114 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:27.370 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:27.305 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:27.374 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:27.309 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:27.386 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:27.295 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:27.387 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:27.323 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:27.436 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:27.371 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:27.443 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:27.352 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:27.467 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:27.403 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:27.470 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:27.405 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:27.491 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:27.399 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:34.033 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:33.968 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:34.039 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:33.975 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:34.065 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:34.001 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:34.089 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:33.998 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:34.092 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:34.001 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:34.106 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:34.041 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:34.109 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:34.045 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:34.112 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:34.021 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:34.133 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:34.069 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:54.176 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:54.111 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:54.240 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:54.176 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:54.292 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:54.227 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:54.297 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:54.206 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:54.298 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:54.207 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:54.306 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:54.241 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:54.308 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:54.244 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:54.340 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:54.249 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:54.349 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:54.284 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:18:57.350 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3295 ms) 2023-10-30T12:18:57.350 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2023-10-30T12:18:57.351 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (8026 ms) 2023-10-30T12:18:57.351 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2023-10-30T12:18:57.351 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7743 ms) 2023-10-30T12:18:57.352 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2023-10-30T12:18:57.352 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7465 ms) 2023-10-30T12:18:57.352 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2023-10-30T12:18:57.353 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7463 ms) 2023-10-30T12:18:57.353 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2023-10-30T12:18:57.353 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6873 ms) 2023-10-30T12:18:57.354 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2023-10-30T12:18:57.354 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7065 ms) 2023-10-30T12:18:57.355 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (146578 ms total) 2023-10-30T12:18:57.355 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: 2023-10-30T12:18:57.355 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [----------] Global test environment tear-down 2023-10-30T12:18:57.356 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (436146 ms total) 2023-10-30T12:18:57.356 INFO:tasks.workunit.client.0.smithi003.stdout: api_aio_pp: [ PASSED ] 57 tests. 2023-10-30T12:19:14.196 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:14.131 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:14.199 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:14.134 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:14.221 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:14.156 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:14.248 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:14.156 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:14.249 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:14.158 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:14.266 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:14.202 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:14.268 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:14.177 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:14.269 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:14.205 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:14.289 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:14.225 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:34.278 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:34.213 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:34.280 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:34.216 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:34.282 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:34.217 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:34.283 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:34.219 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:34.324 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:34.232 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:34.326 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:34.235 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:34.335 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:34.270 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:34.339 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:34.275 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:34.360 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:34.268 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:54.322 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:54.257 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:54.374 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:54.310 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:54.420 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:54.329 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:54.421 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:54.357 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:54.425 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:54.334 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:54.440 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:54.376 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:54.442 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:54.377 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:54.454 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:54.363 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:19:54.474 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:54.410 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:14.508 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:14.443 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:14.512 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:14.448 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:14.545 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:14.481 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:14.560 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:14.469 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:14.561 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:14.496 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:14.562 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:14.470 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:14.565 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:14.500 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:14.593 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:14.502 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:14.597 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:14.533 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:17.957 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2023-10-30T12:20:17.957 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [----------] Global test environment set-up. 2023-10-30T12:20:17.957 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2023-10-30T12:20:17.957 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: seed 92123 2023-10-30T12:20:17.958 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2023-10-30T12:20:17.958 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (22844 ms) 2023-10-30T12:20:17.958 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2023-10-30T12:20:17.958 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (102250 ms) 2023-10-30T12:20:17.959 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2023-10-30T12:20:17.959 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (716 ms) 2023-10-30T12:20:17.959 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (125810 ms total) 2023-10-30T12:20:17.959 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: 2023-10-30T12:20:17.959 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2023-10-30T12:20:17.959 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2023-10-30T12:20:17.960 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (25193 ms) 2023-10-30T12:20:17.960 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2023-10-30T12:20:17.960 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (16081 ms) 2023-10-30T12:20:17.960 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2023-10-30T12:20:17.960 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (29035 ms) 2023-10-30T12:20:17.961 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2023-10-30T12:20:17.961 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: my_snaps [4] 2023-10-30T12:20:17.961 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: my_snaps [5,4] 2023-10-30T12:20:17.961 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: my_snaps [6,5,4] 2023-10-30T12:20:17.961 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: my_snaps [7,6,5,4] 2023-10-30T12:20:17.962 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: promoting some heads 2023-10-30T12:20:17.962 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: promoting from clones for snap 7 2023-10-30T12:20:17.962 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: promoting from clones for snap 6 2023-10-30T12:20:17.962 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: promoting from clones for snap 5 2023-10-30T12:20:17.962 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: promoting from clones for snap 4 2023-10-30T12:20:17.963 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: waiting for scrubs... 2023-10-30T12:20:17.963 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: done waiting 2023-10-30T12:20:17.963 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (141637 ms) 2023-10-30T12:20:17.963 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2023-10-30T12:20:17.963 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (11573 ms) 2023-10-30T12:20:17.964 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2023-10-30T12:20:17.964 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (9131 ms) 2023-10-30T12:20:17.964 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2023-10-30T12:20:17.964 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8972 ms) 2023-10-30T12:20:17.964 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2023-10-30T12:20:17.965 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (9821 ms) 2023-10-30T12:20:17.965 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2023-10-30T12:20:17.965 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (11278 ms) 2023-10-30T12:20:17.965 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2023-10-30T12:20:17.966 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (10733 ms) 2023-10-30T12:20:17.966 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2023-10-30T12:20:17.966 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (12074 ms) 2023-10-30T12:20:17.966 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2023-10-30T12:20:17.966 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (38239 ms) 2023-10-30T12:20:17.967 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2023-10-30T12:20:17.967 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (9911 ms) 2023-10-30T12:20:17.967 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2023-10-30T12:20:17.967 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (10955 ms) 2023-10-30T12:20:17.967 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2023-10-30T12:20:17.968 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (14715 ms) 2023-10-30T12:20:17.968 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2023-10-30T12:20:17.968 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (10137 ms) 2023-10-30T12:20:17.968 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2023-10-30T12:20:17.968 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (11669 ms) 2023-10-30T12:20:28.499 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:28.434 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:28.510 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:28.446 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:28.525 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:28.461 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:28.529 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:28.437 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:28.541 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:28.477 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:28.592 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:28.500 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:28.633 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:28.568 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:28.641 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:28.550 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:28.652 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:28.588 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:34.592 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:34.528 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:34.594 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:34.530 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:34.620 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:34.555 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:34.635 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:34.544 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:34.639 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:34.548 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:34.655 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:34.590 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:34.656 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:34.564 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:34.656 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:34.592 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:34.674 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:34.609 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:39.843 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2023-10-30T12:20:39.843 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: hmm, no HitSet yet 2023-10-30T12:20:39.843 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: ok, hit_set contains 200:602f83fe:::foo:head 2023-10-30T12:20:39.844 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9445 ms) 2023-10-30T12:20:39.844 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2023-10-30T12:20:39.844 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg_num = 32 2023-10-30T12:20:39.844 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 0 ls 1698668435,0 2023-10-30T12:20:39.844 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 1 ls 1698668435,0 2023-10-30T12:20:39.845 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 2 ls 1698668435,0 2023-10-30T12:20:39.845 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 3 ls 1698668435,0 2023-10-30T12:20:39.845 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 4 ls 1698668435,0 2023-10-30T12:20:39.845 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 5 ls 1698668435,0 2023-10-30T12:20:39.845 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 6 ls 1698668435,0 2023-10-30T12:20:39.846 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 7 ls 1698668435,0 2023-10-30T12:20:39.846 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 8 ls 1698668435,0 2023-10-30T12:20:39.846 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 9 ls 1698668435,0 2023-10-30T12:20:39.846 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 10 ls 1698668435,0 2023-10-30T12:20:39.846 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 11 ls 1698668435,0 2023-10-30T12:20:39.847 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 12 ls 1698668435,0 2023-10-30T12:20:39.847 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 13 ls 1698668435,0 2023-10-30T12:20:39.847 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 14 ls 1698668435,0 2023-10-30T12:20:39.847 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 15 ls 1698668435,0 2023-10-30T12:20:39.847 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 16 ls 1698668435,0 2023-10-30T12:20:39.847 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 17 ls 1698668435,0 2023-10-30T12:20:39.848 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 18 ls 1698668435,0 2023-10-30T12:20:39.848 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 19 ls 1698668435,0 2023-10-30T12:20:39.848 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 20 ls 1698668435,0 2023-10-30T12:20:39.848 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 21 ls 1698668435,0 2023-10-30T12:20:39.849 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 22 ls 1698668435,0 2023-10-30T12:20:39.849 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 23 ls 1698668435,0 2023-10-30T12:20:39.849 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 24 ls 1698668435,0 2023-10-30T12:20:39.849 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 25 ls 1698668435,0 2023-10-30T12:20:39.849 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 26 ls 1698668435,0 2023-10-30T12:20:39.850 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 27 ls 1698668435,0 2023-10-30T12:20:39.850 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 28 ls 1698668435,0 2023-10-30T12:20:39.850 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 29 ls 1698668435,0 2023-10-30T12:20:39.850 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 30 ls 1698668435,0 2023-10-30T12:20:39.850 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg 31 ls 1698668435,0 2023-10-30T12:20:39.851 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: pg_num = 32 2023-10-30T12:20:39.851 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:6cac518f:::0:head 2023-10-30T12:20:39.851 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:02547ec2:::1:head 2023-10-30T12:20:39.851 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:f905c69b:::2:head 2023-10-30T12:20:39.851 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:cfc208b3:::3:head 2023-10-30T12:20:39.851 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:d83876eb:::4:head 2023-10-30T12:20:39.852 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:b29083e3:::5:head 2023-10-30T12:20:39.852 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:c4fdafeb:::6:head 2023-10-30T12:20:39.852 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:5c6b0b28:::7:head 2023-10-30T12:20:39.852 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:bd63b0f1:::8:head 2023-10-30T12:20:39.852 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:e960b815:::9:head 2023-10-30T12:20:39.853 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:52ea6a34:::10:head 2023-10-30T12:20:39.853 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:89d3ae78:::11:head 2023-10-30T12:20:39.853 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:de5d7c5f:::12:head 2023-10-30T12:20:39.853 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:566253c9:::13:head 2023-10-30T12:20:39.853 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:62a1935d:::14:head 2023-10-30T12:20:39.854 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:863748b0:::15:head 2023-10-30T12:20:39.854 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:3958e169:::16:head 2023-10-30T12:20:39.854 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:4d4dabf9:::17:head 2023-10-30T12:20:39.854 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:8391935d:::18:head 2023-10-30T12:20:39.855 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:28883081:::19:head 2023-10-30T12:20:39.855 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:69259c59:::20:head 2023-10-30T12:20:39.855 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:4bdb80b7:::21:head 2023-10-30T12:20:39.855 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:a11c5d71:::22:head 2023-10-30T12:20:39.855 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:271af37b:::23:head 2023-10-30T12:20:39.855 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:95b121be:::24:head 2023-10-30T12:20:39.856 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:58d1031b:::25:head 2023-10-30T12:20:39.856 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:0a050783:::26:head 2023-10-30T12:20:39.856 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:c709704c:::27:head 2023-10-30T12:20:39.856 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:cbe56eaf:::28:head 2023-10-30T12:20:39.856 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:86b4b162:::29:head 2023-10-30T12:20:39.857 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:70d89383:::30:head 2023-10-30T12:20:39.857 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:dd450c7c:::31:head 2023-10-30T12:20:39.857 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:6d5729b1:::32:head 2023-10-30T12:20:39.857 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:c388f3fb:::33:head 2023-10-30T12:20:39.857 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:56cfea31:::34:head 2023-10-30T12:20:39.858 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:9dbc1bf7:::35:head 2023-10-30T12:20:39.858 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:40b74ccd:::36:head 2023-10-30T12:20:39.858 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:4d5aaf42:::37:head 2023-10-30T12:20:39.858 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:920f362c:::38:head 2023-10-30T12:20:39.858 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:6cc53222:::39:head 2023-10-30T12:20:39.858 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:9cad833f:::40:head 2023-10-30T12:20:39.859 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:1ea84d41:::41:head 2023-10-30T12:20:39.859 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:c4480ef6:::42:head 2023-10-30T12:20:39.859 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:a694361e:::43:head 2023-10-30T12:20:39.859 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:d1bd33e9:::44:head 2023-10-30T12:20:39.859 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:ddc2cd5d:::45:head 2023-10-30T12:20:39.860 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:2b782207:::46:head 2023-10-30T12:20:39.860 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:7b187fca:::47:head 2023-10-30T12:20:39.860 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:90ecdf6f:::48:head 2023-10-30T12:20:39.860 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:a5ed95fe:::49:head 2023-10-30T12:20:39.860 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:ea0eaa55:::50:head 2023-10-30T12:20:39.861 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:f33ef17b:::51:head 2023-10-30T12:20:39.861 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:a0d1b2f6:::52:head 2023-10-30T12:20:39.861 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:60c5229e:::53:head 2023-10-30T12:20:39.861 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:edcbc575:::54:head 2023-10-30T12:20:39.861 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:102cf253:::55:head 2023-10-30T12:20:39.862 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:efb7fb0b:::56:head 2023-10-30T12:20:39.862 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:50d0a326:::57:head 2023-10-30T12:20:39.862 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:d4dc5daf:::58:head 2023-10-30T12:20:39.862 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:3a130462:::59:head 2023-10-30T12:20:39.862 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:ec87ed71:::60:head 2023-10-30T12:20:39.863 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:d5bc9454:::61:head 2023-10-30T12:20:39.863 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:3ddfe313:::62:head 2023-10-30T12:20:39.863 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:7c2816b9:::63:head 2023-10-30T12:20:39.863 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:47e00e4d:::64:head 2023-10-30T12:20:39.863 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:c6410c18:::65:head 2023-10-30T12:20:39.863 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:b48ed237:::66:head 2023-10-30T12:20:39.864 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:cd63ad31:::67:head 2023-10-30T12:20:39.864 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:b179e92b:::68:head 2023-10-30T12:20:39.864 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:0d9f741a:::69:head 2023-10-30T12:20:39.864 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:6d3352ae:::70:head 2023-10-30T12:20:39.864 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:c6d5c19e:::71:head 2023-10-30T12:20:39.865 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:bc4729c3:::72:head 2023-10-30T12:20:39.865 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:77e930b9:::73:head 2023-10-30T12:20:39.865 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:0abeecfd:::74:head 2023-10-30T12:20:39.865 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:b7c37e15:::75:head 2023-10-30T12:20:39.865 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:b6378398:::76:head 2023-10-30T12:20:39.865 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:02bd68de:::77:head 2023-10-30T12:20:39.866 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:cc795d2d:::78:head 2023-10-30T12:20:39.866 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:630d4fea:::79:head 2023-10-30T12:20:39.866 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:e0d29ef5:::80:head 2023-10-30T12:20:39.866 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:fd6f13d2:::81:head 2023-10-30T12:20:39.866 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:606461d5:::82:head 2023-10-30T12:20:39.867 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:eadbdc43:::83:head 2023-10-30T12:20:39.867 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:8761d0bb:::84:head 2023-10-30T12:20:39.867 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:9ef0186f:::85:head 2023-10-30T12:20:39.867 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:e0d41294:::86:head 2023-10-30T12:20:39.867 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:961de695:::87:head 2023-10-30T12:20:39.868 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:1423148f:::88:head 2023-10-30T12:20:39.868 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:633a8fa2:::89:head 2023-10-30T12:20:39.868 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:a8653809:::90:head 2023-10-30T12:20:39.868 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:3dac8b33:::91:head 2023-10-30T12:20:39.868 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:35aad435:::92:head 2023-10-30T12:20:39.868 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:f6dcc343:::93:head 2023-10-30T12:20:39.869 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:dbbdad87:::94:head 2023-10-30T12:20:39.869 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:1cb48ce0:::95:head 2023-10-30T12:20:39.869 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:03cd461c:::96:head 2023-10-30T12:20:39.869 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:17a4ea99:::97:head 2023-10-30T12:20:39.869 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:9993c9a7:::98:head 2023-10-30T12:20:39.870 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:6394211c:::99:head 2023-10-30T12:20:39.870 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:94c7ae57:::100:head 2023-10-30T12:20:39.870 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:6fdee5bb:::101:head 2023-10-30T12:20:39.870 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:9a477fd1:::102:head 2023-10-30T12:20:39.870 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:eb850916:::103:head 2023-10-30T12:20:39.871 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:affc56b9:::104:head 2023-10-30T12:20:39.871 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:b42dc814:::105:head 2023-10-30T12:20:39.871 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:f319f8f0:::106:head 2023-10-30T12:20:39.871 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:9a40b9de:::107:head 2023-10-30T12:20:39.871 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:8b524f28:::108:head 2023-10-30T12:20:39.872 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:e3de589f:::109:head 2023-10-30T12:20:39.872 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:90f90a5b:::110:head 2023-10-30T12:20:39.872 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:a7b4f1d7:::111:head 2023-10-30T12:20:39.872 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:af51766e:::112:head 2023-10-30T12:20:39.872 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:b6f90bd1:::113:head 2023-10-30T12:20:39.872 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:e0261208:::114:head 2023-10-30T12:20:39.873 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:c9569ef7:::115:head 2023-10-30T12:20:39.873 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:61bebe50:::116:head 2023-10-30T12:20:39.873 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:fe93412b:::117:head 2023-10-30T12:20:39.873 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:d3d38bee:::118:head 2023-10-30T12:20:39.873 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:3100ba0c:::119:head 2023-10-30T12:20:39.873 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:d0560ada:::120:head 2023-10-30T12:20:39.874 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:f0ea8b35:::121:head 2023-10-30T12:20:39.874 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:766f231a:::122:head 2023-10-30T12:20:39.874 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:a07a2582:::123:head 2023-10-30T12:20:39.874 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:bd7c6b3a:::124:head 2023-10-30T12:20:39.874 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:fb2ddaff:::125:head 2023-10-30T12:20:39.875 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:4408e1fe:::126:head 2023-10-30T12:20:39.875 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:ee1df7a7:::127:head 2023-10-30T12:20:39.875 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:c3002909:::128:head 2023-10-30T12:20:39.875 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:4f48ffa9:::129:head 2023-10-30T12:20:39.875 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:edf38733:::130:head 2023-10-30T12:20:39.876 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:c08425c0:::131:head 2023-10-30T12:20:39.876 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:5f902d98:::132:head 2023-10-30T12:20:39.876 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:41ea2c93:::133:head 2023-10-30T12:20:39.876 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:813cee13:::134:head 2023-10-30T12:20:39.876 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:0131818d:::135:head 2023-10-30T12:20:39.877 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:26ba5a85:::136:head 2023-10-30T12:20:39.877 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:381b8a5a:::137:head 2023-10-30T12:20:39.877 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:28797e47:::138:head 2023-10-30T12:20:39.877 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:bfca7f22:::139:head 2023-10-30T12:20:39.877 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:36807075:::140:head 2023-10-30T12:20:39.877 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:80b03975:::141:head 2023-10-30T12:20:39.878 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:5c15709b:::142:head 2023-10-30T12:20:39.878 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:f39ea15e:::143:head 2023-10-30T12:20:39.878 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:ea992956:::144:head 2023-10-30T12:20:39.878 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:48887b1c:::145:head 2023-10-30T12:20:39.878 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:9f24a9dd:::146:head 2023-10-30T12:20:39.879 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:987f100b:::147:head 2023-10-30T12:20:39.879 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:d2dd3581:::148:head 2023-10-30T12:20:39.879 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:7fed1808:::149:head 2023-10-30T12:20:39.879 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:c80b70e9:::150:head 2023-10-30T12:20:39.879 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:85ed90f9:::151:head 2023-10-30T12:20:39.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:36428b24:::152:head 2023-10-30T12:20:39.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:d044c34a:::153:head 2023-10-30T12:20:39.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:7c18bf58:::154:head 2023-10-30T12:20:39.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:d1c21232:::155:head 2023-10-30T12:20:39.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:a7a3c575:::156:head 2023-10-30T12:20:39.880 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:87da0633:::157:head 2023-10-30T12:20:39.881 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:d5ac3822:::158:head 2023-10-30T12:20:39.881 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:3f20522d:::159:head 2023-10-30T12:20:39.881 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:6ca26563:::160:head 2023-10-30T12:20:39.881 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:532ce135:::161:head 2023-10-30T12:20:39.881 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:c78863e6:::162:head 2023-10-30T12:20:54.666 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:54.601 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:54.670 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:54.606 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:54.698 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:54.634 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:54.716 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:54.624 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:54.719 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:54.627 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:54.734 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:54.669 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:54.737 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:54.673 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:54.743 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:54.652 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:20:54.773 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:54.708 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:19.868 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:19.803 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:19.873 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:19.809 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:19.902 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:19.838 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:19.929 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:19.865 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:19.931 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:19.866 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:19.933 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:19.842 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:19.951 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:19.860 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:19.962 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:19.897 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:19.983 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:19.892 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:34.941 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:34.876 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:34.943 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:34.878 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:34.972 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:34.908 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:34.994 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:34.902 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:34.995 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:34.904 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:34.998 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:34.933 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:34.999 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:34.935 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:35.020 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:34.929 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:35.029 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:34.965 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:39.832 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:39.767 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:39.834 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:39.769 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:39.851 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:39.786 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:39.882 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:39.790 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:39.884 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:39.820 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:39.884 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:39.793 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:39.887 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:39.823 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:39.896 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:39.805 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:39.899 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:39.835 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:47.566 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:7462ddf6:::163:head 2023-10-30T12:21:47.567 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:5d165639:::164:head 2023-10-30T12:21:47.567 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:f43765fc:::165:head 2023-10-30T12:21:47.567 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:b4c720e9:::166:head 2023-10-30T12:21:47.567 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:e694b040:::167:head 2023-10-30T12:21:47.568 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:afa38db2:::168:head 2023-10-30T12:21:47.568 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:77ba9f53:::169:head 2023-10-30T12:21:47.568 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:87495034:::170:head 2023-10-30T12:21:47.568 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:7c96bf0e:::171:head 2023-10-30T12:21:47.568 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:dbe346cc:::172:head 2023-10-30T12:21:47.569 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:e943ec24:::173:head 2023-10-30T12:21:47.569 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:f97a9c0c:::174:head 2023-10-30T12:21:47.569 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:6f26e74d:::175:head 2023-10-30T12:21:47.569 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:4f95e106:::176:head 2023-10-30T12:21:47.569 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:0e6f2f8f:::177:head 2023-10-30T12:21:47.569 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:05db05f1:::178:head 2023-10-30T12:21:47.570 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:38a78d66:::179:head 2023-10-30T12:21:47.570 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:d095610b:::180:head 2023-10-30T12:21:47.570 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:a1a9d709:::181:head 2023-10-30T12:21:47.570 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:1e5d39db:::182:head 2023-10-30T12:21:47.570 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:f7df4fb9:::183:head 2023-10-30T12:21:47.571 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:03a7f161:::184:head 2023-10-30T12:21:47.571 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:ba70721e:::185:head 2023-10-30T12:21:47.571 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:28e5662d:::186:head 2023-10-30T12:21:47.571 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:973d52de:::187:head 2023-10-30T12:21:47.571 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:4303eb1c:::188:head 2023-10-30T12:21:47.572 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:b990b48e:::189:head 2023-10-30T12:21:47.572 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:29b8165b:::190:head 2023-10-30T12:21:47.572 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:3547f197:::191:head 2023-10-30T12:21:47.572 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:7e260936:::192:head 2023-10-30T12:21:47.572 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:1abec7b1:::193:head 2023-10-30T12:21:47.573 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:10fdda93:::194:head 2023-10-30T12:21:47.573 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:15817eea:::195:head 2023-10-30T12:21:47.573 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:770bab57:::196:head 2023-10-30T12:21:47.573 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:ed9e13e7:::197:head 2023-10-30T12:21:47.573 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:71471a8f:::198:head 2023-10-30T12:21:47.574 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: checking for 201:10fb1d02:::199:head 2023-10-30T12:21:47.574 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (22140 ms) 2023-10-30T12:21:47.574 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2023-10-30T12:21:47.574 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698668457,0 2023-10-30T12:21:47.574 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: first is 1698668457 2023-10-30T12:21:47.575 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698668457,0 2023-10-30T12:21:47.575 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698668457,0 2023-10-30T12:21:47.575 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698668457,1698668459,1698668460,0 2023-10-30T12:21:47.575 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698668457,1698668459,1698668460,0 2023-10-30T12:21:47.575 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698668457,1698668459,1698668460,0 2023-10-30T12:21:47.576 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698668457,1698668459,1698668460,1698668462,1698668463,0 2023-10-30T12:21:47.576 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698668457,1698668459,1698668460,1698668462,1698668463,0 2023-10-30T12:21:47.576 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698668457,1698668459,1698668460,1698668462,1698668463,0 2023-10-30T12:21:47.576 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698668457,1698668459,1698668460,1698668462,1698668463,1698668465,1698668466,0 2023-10-30T12:21:47.576 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698668457,1698668459,1698668460,1698668462,1698668463,1698668465,1698668466,0 2023-10-30T12:21:47.577 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698668457,1698668459,1698668460,1698668462,1698668463,1698668465,1698668466,0 2023-10-30T12:21:47.577 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698668460,1698668462,1698668463,1698668465,1698668466,1698668469,1698668470,0 2023-10-30T12:21:47.577 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: first now 1698668460, trimmed 2023-10-30T12:21:47.577 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22309 ms) 2023-10-30T12:21:47.577 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2023-10-30T12:21:47.578 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: foo0 2023-10-30T12:21:47.578 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-10-30T12:21:47.578 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14682 ms) 2023-10-30T12:21:47.578 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2023-10-30T12:21:50.212 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:50.147 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:50.216 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:50.151 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:50.246 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:50.181 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:50.270 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:50.179 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:50.279 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:50.187 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:50.285 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:50.220 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:50.292 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:50.228 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:50.296 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:50.205 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:50.318 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:50.254 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:59.978 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:59.913 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:21:59.980 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:59.916 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:00.012 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:59.947 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:00.038 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:59.946 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:00.038 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:59.974 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:00.040 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:59.949 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:00.040 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:59.976 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:00.061 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:59.970 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:00.063 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:59.999 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:04.992 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:04.928 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:04.995 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:04.931 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:05.012 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:04.947 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:05.044 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:04.953 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:05.046 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:04.955 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:05.047 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:04.982 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:05.050 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:04.985 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:05.057 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:04.966 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:05.064 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:04.999 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.024 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:19.959 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.085 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:20.019 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.125 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:20.060 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.132 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:20.041 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.134 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:20.043 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.146 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:20.081 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.148 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:20.083 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.172 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:20.081 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.175 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:20.110 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.853 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:20.761 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.854 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:20.762 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.862 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:20.771 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.947 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:20.882 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.948 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:20.883 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.957 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:20.892 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.976 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:20.911 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.977 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:20.912 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:20.985 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:20.921 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:35.118 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:35.053 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:35.122 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:35.058 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:35.149 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:35.084 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:35.170 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:35.105 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:35.171 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:35.080 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:35.173 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:35.081 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:35.173 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:35.108 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:35.201 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:35.136 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:35.205 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:35.114 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:40.127 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:40.063 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:40.132 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:40.067 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:40.148 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:40.084 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:40.175 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:40.083 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:40.178 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:40.087 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:40.181 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:40.116 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:40.184 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:40.119 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:40.192 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:40.101 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:40.199 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:40.134 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:50.270 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:50.204 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:50.271 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:50.206 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:50.291 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:13:50.227 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:50.317 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:50.226 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:50.319 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:50.227 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:50.319 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:50.254 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:50.320 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:50.256 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:50.340 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:13:50.276 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:22:50.361 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:50.269 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:00.318 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:00.253 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:00.320 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:00.256 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:00.333 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:00.269 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:00.371 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:00.306 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:00.372 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:00.308 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:00.374 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:00.282 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:00.375 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:00.284 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:00.390 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:00.326 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:00.420 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:00.328 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:10.248 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:10.183 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:10.249 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:10.185 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:10.263 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:10.199 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:10.286 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:10.195 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:10.289 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:10.198 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:10.297 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:10.233 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:10.300 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:10.236 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:10.302 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:10.211 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:10.314 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:10.250 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:25.325 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:25.260 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:25.327 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:25.263 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:25.350 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:25.286 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:25.370 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:25.279 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:25.372 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:25.281 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:25.395 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:25.303 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:25.399 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:25.334 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:25.400 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:25.336 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:25.473 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:25.408 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:25.984 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:25.892 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:25.984 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:25.893 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:25.996 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:25.904 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:26.037 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:25.972 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:26.038 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:25.973 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:26.046 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:25.982 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:26.058 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:25.993 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:26.059 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:25.995 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:26.071 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:26.006 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:45.601 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:45.536 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:45.611 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:45.547 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:45.660 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:45.595 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:45.674 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:45.583 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:45.677 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:45.586 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:45.690 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:45.626 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:45.695 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:45.630 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:45.713 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:45.622 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:45.753 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:45.688 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:46.500 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:46.408 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:46.500 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:46.409 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:46.511 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:46.420 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:46.565 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:46.500 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:46.566 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:46.501 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:46.566 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:46.502 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:46.567 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:46.503 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:46.577 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:14:46.512 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:23:46.581 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:14:46.516 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:05.592 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:05.528 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:05.645 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:05.581 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:05.668 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:05.604 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:05.693 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:05.602 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:05.696 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:05.605 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:05.716 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:05.624 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:05.718 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:05.654 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:05.721 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:05.657 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:05.743 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:05.678 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:06.224 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:06.133 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:06.225 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:06.134 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:06.233 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:06.141 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:06.275 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:06.211 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:06.276 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:06.212 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:06.282 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:06.217 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:06.283 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:06.218 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:06.284 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:06.220 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:06.291 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:06.227 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:25.701 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:25.636 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:25.705 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:25.640 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:25.743 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:25.678 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:25.750 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:25.659 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:25.754 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:25.663 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:25.769 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:25.705 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:25.771 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:25.707 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:25.776 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:25.685 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:25.794 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:25.730 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:26.177 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:26.085 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:26.178 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:26.086 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:26.185 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:26.094 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:26.241 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:26.177 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:26.244 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:26.179 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:26.256 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:26.191 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:26.258 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:26.193 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:26.259 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:26.194 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:26.267 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:26.202 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:40.899 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:40.834 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:40.907 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:40.843 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:40.931 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:40.866 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:40.960 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:40.869 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:40.962 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:40.871 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:40.973 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:40.909 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:40.975 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:40.910 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:41.004 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:40.939 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:41.011 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:40.919 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:45.772 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:45.707 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:45.774 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:45.709 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:45.787 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:45.723 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:45.820 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:45.728 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:45.821 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:45.730 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:45.833 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:45.742 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:45.837 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:45.773 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:45.840 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:45.775 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:45.857 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:45.792 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:55.801 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:55.737 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:55.804 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:55.739 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:55.817 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:15:55.752 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:55.852 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:55.787 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:55.853 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:55.762 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:55.854 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:55.789 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:55.855 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:55.764 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:55.868 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:55.776 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:24:55.872 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:15:55.807 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:05.832 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:05.767 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:05.834 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:05.769 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:05.847 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:05.783 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:05.884 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:05.792 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:05.885 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:05.794 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:05.886 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:05.822 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:05.888 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:05.824 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:05.897 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:05.805 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:05.900 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:05.835 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:05.978 INFO:tasks.ceph.osd.1.smithi003.stderr:2023-10-30T12:25:05.953+0000 22288640 -1 snap_mapper.add_oid found existing snaps mapped on 219:48337271:test-rados-api-smithi003-92123-80::foo:2, removing 2023-10-30T12:25:05.989 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-10-30T12:25:05.981+0000 29296640 -1 snap_mapper.add_oid found existing snaps mapped on 219:48337271:test-rados-api-smithi003-92123-80::foo:2, removing 2023-10-30T12:25:13.493 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:13.428 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:13.504 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:13.440 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:13.509 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:13.444 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:13.518 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:13.427 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:13.531 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:13.466 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:13.574 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:13.509 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:13.576 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:13.484 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:13.602 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:13.538 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:13.606 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:13.515 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:16.010 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:15.945 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:16.012 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:15.947 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:16.029 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:15.964 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:16.061 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:15.970 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:16.065 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:15.974 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:16.075 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:15.984 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:16.076 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:16.011 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:16.077 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:16.013 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:16.089 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:16.024 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:25.915 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:25.850 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:25.917 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:25.852 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:25.950 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:25.885 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:25.959 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:25.868 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:25.963 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:25.872 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:25.977 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:25.913 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:25.978 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:25.914 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:25.984 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:25.892 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:25.999 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:25.934 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:30.947 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:30.882 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:30.952 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:30.888 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:30.979 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:30.914 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:31.011 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:30.920 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:31.013 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:30.922 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:31.029 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:30.964 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:31.029 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:30.938 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:31.031 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:30.966 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:31.051 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:30.986 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:42.816 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (21034 ms) 2023-10-30T12:25:42.816 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2023-10-30T12:25:42.817 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (24509 ms) 2023-10-30T12:25:42.817 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2023-10-30T12:25:42.817 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (4492 ms) 2023-10-30T12:25:42.817 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2023-10-30T12:25:42.817 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:25:42.818 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (4835 ms) 2023-10-30T12:25:42.818 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2023-10-30T12:25:42.818 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4773 ms) 2023-10-30T12:25:42.818 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2023-10-30T12:25:42.818 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:25:42.819 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4959 ms) 2023-10-30T12:25:42.819 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2023-10-30T12:25:42.819 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:25:42.819 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5513 ms) 2023-10-30T12:25:42.819 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2023-10-30T12:25:42.820 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:25:42.820 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40875 ms) 2023-10-30T12:25:42.820 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2023-10-30T12:25:42.820 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:25:42.820 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17858 ms) 2023-10-30T12:25:42.821 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2023-10-30T12:25:42.821 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:25:42.821 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5074 ms) 2023-10-30T12:25:42.821 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2023-10-30T12:25:42.821 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:25:42.821 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4417 ms) 2023-10-30T12:25:42.822 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2023-10-30T12:25:42.822 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:25:42.822 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (27399 ms) 2023-10-30T12:25:42.822 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2023-10-30T12:25:42.822 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:25:42.823 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (17144 ms) 2023-10-30T12:25:42.823 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2023-10-30T12:25:42.823 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:25:42.823 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6396 ms) 2023-10-30T12:25:42.823 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2023-10-30T12:25:42.823 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:25:42.824 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5039 ms) 2023-10-30T12:25:42.824 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2023-10-30T12:25:42.824 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:25:42.824 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: waiting for scrubs... 2023-10-30T12:25:42.824 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: done waiting 2023-10-30T12:25:42.825 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26218 ms) 2023-10-30T12:25:42.825 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2023-10-30T12:25:42.825 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:25:42.825 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (12313 ms) 2023-10-30T12:25:42.825 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2023-10-30T12:25:42.825 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:25:42.826 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (10274 ms) 2023-10-30T12:25:42.826 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2023-10-30T12:25:42.826 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:25:42.826 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (11058 ms) 2023-10-30T12:25:42.826 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2023-10-30T12:25:46.078 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:46.013 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:46.081 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:46.016 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:46.112 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:16:46.047 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:46.133 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:46.042 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:46.135 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:46.044 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:46.141 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:46.077 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:46.143 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:46.078 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:46.163 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:46.071 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:25:46.180 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:16:46.116 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:01.037 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:00.973 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:01.042 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:00.978 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:01.070 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:01.005 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:01.091 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:01.000 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:01.093 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:01.029 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:01.094 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:01.003 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:01.096 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:01.031 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:01.120 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:01.029 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:01.136 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:01.071 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:06.267 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:06.202 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:06.270 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:06.206 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:06.292 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:06.228 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:06.330 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:06.265 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:06.331 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:06.240 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:06.336 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:06.244 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:06.337 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:06.273 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:06.355 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:06.264 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:06.359 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:06.294 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:21.235 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:21.170 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:21.239 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:21.174 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:21.258 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:21.193 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:21.290 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:21.225 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:21.291 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:21.226 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:21.291 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:21.199 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:21.292 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:21.201 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:21.309 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:21.244 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:21.309 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:21.218 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:26.134 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:26.070 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:26.137 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:26.072 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:26.147 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:26.083 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:26.181 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:26.090 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:26.183 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:26.118 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:26.184 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:26.092 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:26.188 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:26.123 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:26.194 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:26.103 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:26.199 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:26.134 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:36.177 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:36.112 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:36.181 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:36.117 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:36.201 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:36.137 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:36.228 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:36.136 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:36.229 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:36.138 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:36.233 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:36.169 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:36.236 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:36.171 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:36.247 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:36.155 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:36.257 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:36.192 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:43.817 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-10-30T12:26:43.801+0000 21a87640 -1 snap_mapper.add_oid found existing snaps mapped on 73:1a6c4dc6:test-rados-api-smithi003-92123-97::foo:31, removing 2023-10-30T12:26:43.827 INFO:tasks.ceph.osd.3.smithi003.stderr:2023-10-30T12:26:43.818+0000 29296640 -1 snap_mapper.add_oid found existing snaps mapped on 73:1a6c4dc6:test-rados-api-smithi003-92123-97::foo:31, removing 2023-10-30T12:26:46.278 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:46.213 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:46.327 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:46.262 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:46.330 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:46.266 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:46.338 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:46.274 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:46.353 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:46.289 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:46.366 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:46.302 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:46.385 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:46.294 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:46.395 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:46.303 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:46.414 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:46.322 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:51.237 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:51.172 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:51.238 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:51.173 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:51.251 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:17:51.187 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:51.283 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:51.192 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:51.284 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:51.193 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:51.287 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:51.223 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:51.289 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:51.225 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:51.296 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:51.205 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:26:51.304 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:17:51.239 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:06.476 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:06.411 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:06.478 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:06.413 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:06.507 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:06.442 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:06.527 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:06.435 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:06.528 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:06.464 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:06.530 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:06.465 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:06.530 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:06.438 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:06.561 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:06.470 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:06.562 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:06.497 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:16.704 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:16.639 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:16.707 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:16.643 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:16.733 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:16.669 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:16.759 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:16.668 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:16.761 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:16.670 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:16.761 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:16.697 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:16.766 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:16.701 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:16.787 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:16.695 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:16.791 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:16.726 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:21.403 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:21.339 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:21.407 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:21.342 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:21.420 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:21.355 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:21.450 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:21.359 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:21.454 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:21.363 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:21.455 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:21.391 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:21.457 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:21.393 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:21.469 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:21.405 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:21.471 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:21.380 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:26.414 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:26.350 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:26.416 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:26.352 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:26.437 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:26.372 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:26.459 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:26.368 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:26.461 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:26.369 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:26.470 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:26.405 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:26.472 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:26.408 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:26.475 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:26.383 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:26.488 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:26.423 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:36.411 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:36.347 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:36.464 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:36.399 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:36.471 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:36.407 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:36.475 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:36.411 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:36.497 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:36.432 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:36.509 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:36.445 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:36.511 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:36.419 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:36.516 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:36.425 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:36.538 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:36.447 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:46.503 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:46.437 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:46.505 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:46.440 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:46.539 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:46.474 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:46.547 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:46.456 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:46.550 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:46.459 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:46.558 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:46.494 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:46.560 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:46.495 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:46.573 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:46.482 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:46.584 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:46.519 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:51.573 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:51.508 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:51.635 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:51.570 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:51.657 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:18:51.592 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:51.688 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:51.597 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:51.690 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:51.598 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:51.697 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:51.632 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:51.699 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:51.634 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:51.706 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:51.615 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:27:51.716 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:18:51.651 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:06.581 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:06.516 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:06.582 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:06.518 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:06.625 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:06.560 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:06.625 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:06.534 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:06.628 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:06.537 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:06.643 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:06.579 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:06.646 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:06.582 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:06.656 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:06.564 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:06.676 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:06.612 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:07.246 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:07.154 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:07.247 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:07.155 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:07.255 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:07.163 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:07.377 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:07.313 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:07.379 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:07.314 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:07.384 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:07.320 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:07.386 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:07.321 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:07.391 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:07.326 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:07.395 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:07.331 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:21.733 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:21.668 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:21.735 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:21.671 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:21.762 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:21.697 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:21.780 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:21.689 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:21.782 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:21.717 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:21.782 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:21.690 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:21.783 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:21.719 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:21.808 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:21.716 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:21.824 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:21.759 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:26.747 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:26.682 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:26.749 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:26.684 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:26.760 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:26.669 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:26.763 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:26.672 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:26.766 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:26.701 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:26.775 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:26.683 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:26.796 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:26.732 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:26.798 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:26.733 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:26.811 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:26.746 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:36.690 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:36.624 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:36.697 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:36.633 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:36.717 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:36.652 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:36.742 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:36.650 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:36.744 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:36.653 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:36.755 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:36.664 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:36.767 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:36.702 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:36.769 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:36.704 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:36.781 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:36.717 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:51.842 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:51.777 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:51.846 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:51.781 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:51.875 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:19:51.811 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:51.893 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:51.802 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:51.894 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:51.803 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:51.897 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:51.832 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:51.902 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:51.837 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:51.918 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:51.827 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:28:51.934 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:19:51.870 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:06.820 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:06.755 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:06.827 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:06.762 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:06.875 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:06.810 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:06.890 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:06.799 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:06.896 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:06.832 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:06.897 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:06.805 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:06.902 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:06.838 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:06.940 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:06.849 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:06.958 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:06.893 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:11.971 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:11.906 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:11.977 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:11.913 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:12.001 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:11.937 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:12.024 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:11.933 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:12.027 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:11.935 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:12.040 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:11.975 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:12.040 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:11.949 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:12.042 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:11.977 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:12.059 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:11.995 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:13.128 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:13.063 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:13.132 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:13.067 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:13.145 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:13.081 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:13.190 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:13.125 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:13.190 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:13.099 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:13.192 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:13.127 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:13.194 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:13.103 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:13.208 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:13.116 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:13.208 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:13.143 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:21.885 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:21.820 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:21.888 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:21.823 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:21.906 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:21.842 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:21.929 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:21.838 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:21.931 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:21.839 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:21.934 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:21.870 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:21.937 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:21.873 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:21.950 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:21.859 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:21.961 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:21.896 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:32.009 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:31.945 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:32.058 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:31.993 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:32.082 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:32.018 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:32.103 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:32.011 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:32.106 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:32.015 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:32.109 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:32.045 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:32.112 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:32.047 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:32.128 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:32.036 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:32.134 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:32.070 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:37.023 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:29:37.024 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9575 ms) 2023-10-30T12:29:37.024 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2023-10-30T12:29:37.024 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:29:37.024 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (7170 ms) 2023-10-30T12:29:37.024 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2023-10-30T12:29:37.025 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:29:37.025 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6570 ms) 2023-10-30T12:29:37.025 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2023-10-30T12:29:37.025 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:29:37.026 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26594 ms) 2023-10-30T12:29:37.026 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2023-10-30T12:29:37.026 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:29:37.026 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14895 ms) 2023-10-30T12:29:37.026 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2023-10-30T12:29:37.027 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (11950 ms) 2023-10-30T12:29:37.027 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2023-10-30T12:29:37.027 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2023-10-30T12:29:37.027 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: 00000010 69 73 |is| 2023-10-30T12:29:37.027 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: 00000012 2023-10-30T12:29:37.028 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (13079 ms) 2023-10-30T12:29:37.028 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2023-10-30T12:29:37.028 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:29:37.028 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6986 ms) 2023-10-30T12:29:37.028 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (800730 ms total) 2023-10-30T12:29:37.029 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: 2023-10-30T12:29:37.029 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2023-10-30T12:29:37.029 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2023-10-30T12:29:37.029 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1470 ms) 2023-10-30T12:29:37.029 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2023-10-30T12:29:37.030 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (12288 ms) 2023-10-30T12:29:37.030 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2023-10-30T12:29:37.030 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (17734 ms) 2023-10-30T12:29:37.030 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2023-10-30T12:29:37.031 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (18 ms) 2023-10-30T12:29:37.031 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (31510 ms total) 2023-10-30T12:29:37.031 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: 2023-10-30T12:29:37.031 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2023-10-30T12:29:37.031 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2023-10-30T12:29:37.032 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7861 ms) 2023-10-30T12:29:37.032 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2023-10-30T12:29:37.032 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8808 ms) 2023-10-30T12:29:37.032 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2023-10-30T12:29:37.032 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: waiting for scrub... 2023-10-30T12:29:37.033 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: done waiting 2023-10-30T12:29:37.033 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (26280 ms) 2023-10-30T12:29:37.033 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2023-10-30T12:29:37.033 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (12150 ms) 2023-10-30T12:29:37.033 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2023-10-30T12:29:37.033 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (9237 ms) 2023-10-30T12:29:37.034 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2023-10-30T12:29:37.034 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (10110 ms) 2023-10-30T12:29:37.034 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2023-10-30T12:29:37.034 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (12548 ms) 2023-10-30T12:29:37.034 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2023-10-30T12:29:37.082 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:37.018 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:37.085 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:37.020 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:37.098 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:37.033 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:37.132 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:37.067 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:37.133 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:37.069 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:37.140 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:37.048 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:37.141 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:37.050 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:37.149 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:37.084 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:37.155 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:37.064 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:52.228 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:52.163 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:52.231 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:52.166 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:52.242 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:52.177 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:52.262 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:20:52.198 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:52.302 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:52.237 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:52.303 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:52.212 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:52.305 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:52.213 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:52.337 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:20:52.272 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:29:52.338 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:52.246 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:07.099 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:07.034 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:07.101 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:07.037 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:07.132 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:07.068 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:07.137 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:07.072 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:07.149 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:07.057 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:07.151 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:07.060 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:07.157 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:07.092 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:07.179 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:07.087 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:07.186 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:07.121 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:12.300 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:12.235 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:12.302 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:12.237 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:12.316 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:12.252 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:12.346 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:12.255 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:12.349 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:12.257 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:12.350 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:12.285 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:12.351 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:12.287 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:12.363 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:12.272 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:12.366 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:12.301 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:22.309 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:22.244 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:22.312 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:22.247 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:22.347 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:22.282 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:22.363 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:22.271 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:22.364 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:22.273 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:22.371 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:22.307 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:22.376 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:22.312 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:22.385 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:22.294 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:22.399 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:22.335 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:32.394 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:32.329 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:32.396 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:32.331 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:32.433 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:32.368 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:32.452 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:32.387 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:32.456 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:32.365 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:32.457 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:32.392 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:32.458 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:32.367 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:32.481 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:32.416 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:32.493 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:32.402 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:37.310 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:37.245 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:37.312 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:37.247 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:37.325 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:37.261 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:37.355 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:37.290 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:37.359 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:37.295 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:37.360 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:37.269 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:37.367 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:37.275 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:37.375 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:37.311 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:37.379 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:37.288 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:52.451 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:52.386 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:52.499 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:52.435 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:52.502 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:52.438 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:52.505 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:52.440 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:52.531 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:52.467 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:52.535 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:52.470 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:52.561 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:52.470 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:52.562 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:52.471 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:52.586 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:52.495 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:57.414 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:57.349 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:57.418 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:57.354 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:57.430 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:21:57.365 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:57.469 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:57.404 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:57.470 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:57.379 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:57.471 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:57.406 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:57.476 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:57.385 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:57.480 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:21:57.416 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:30:57.487 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:57.395 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:12.396 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:12.331 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:12.444 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:12.353 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:12.445 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:12.354 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:12.447 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:12.382 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:12.449 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:12.385 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:12.450 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:12.386 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:12.468 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:12.377 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:12.477 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:12.412 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:12.486 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:12.421 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:16.439 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:16.374 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:16.445 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:16.380 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:16.462 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:16.398 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:16.493 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:16.402 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:16.497 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:16.406 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:16.501 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:16.437 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:16.503 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:16.438 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:16.514 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:16.423 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:16.520 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:16.455 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:17.614 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:17.550 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:17.621 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:17.556 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:17.632 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:17.567 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:17.674 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:17.609 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:17.675 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:17.584 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:17.676 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:17.611 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:17.681 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:17.589 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:17.690 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:17.625 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:17.693 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:17.601 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:27.586 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:27.522 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:27.587 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:27.523 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:27.609 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:27.544 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:27.631 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:27.539 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:27.631 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:27.566 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:27.632 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:27.540 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:27.634 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:27.570 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:27.651 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:27.560 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:27.666 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:27.601 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:37.578 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:37.513 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:37.582 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:37.517 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:37.601 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:37.536 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:37.627 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:37.536 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:37.633 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:37.568 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:37.635 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:37.544 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:37.636 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:37.571 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:37.647 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:37.556 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:37.654 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:37.589 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:42.708 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:42.643 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:42.711 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:42.647 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:42.727 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:42.663 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:42.731 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:42.666 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:42.734 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:42.670 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:42.755 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:42.691 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:42.771 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:42.680 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:42.773 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:42.682 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:42.788 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:42.697 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:57.862 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:57.797 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:57.864 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:57.800 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:57.894 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:22:57.829 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:57.916 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:57.825 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:57.917 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:57.852 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:57.919 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:57.828 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:57.920 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:57.855 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:57.943 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:57.851 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:31:57.945 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:22:57.881 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:12.981 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:12.916 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:12.991 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:12.927 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:13.018 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:12.953 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:13.044 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:12.953 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:13.046 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:12.955 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:13.053 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:12.989 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:13.056 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:12.991 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:13.074 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:12.982 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:13.084 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:13.020 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:17.799 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:17.734 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:17.801 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:17.737 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:17.814 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:17.749 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:17.846 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:17.781 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:17.849 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:17.784 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:17.859 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:17.767 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:17.861 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:17.769 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:17.867 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:17.803 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:17.875 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:17.784 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:27.976 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:27.912 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:27.983 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:27.919 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.011 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:27.946 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.035 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:27.943 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.037 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:27.945 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.039 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:27.975 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.041 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:27.977 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.113 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:28.022 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.125 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:28.060 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.668 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (10048 ms) 2023-10-30T12:32:28.668 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2023-10-30T12:32:28.668 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14860 ms) 2023-10-30T12:32:28.668 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2023-10-30T12:32:28.669 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (9369 ms) 2023-10-30T12:32:28.669 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2023-10-30T12:32:28.669 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (14351 ms) 2023-10-30T12:32:28.669 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2023-10-30T12:32:28.670 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8330 ms) 2023-10-30T12:32:28.670 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2023-10-30T12:32:28.670 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (9654 ms) 2023-10-30T12:32:28.670 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2023-10-30T12:32:28.670 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: hmm, no HitSet yet 2023-10-30T12:32:28.671 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: ok, hit_set contains 251:602f83fe:::foo:head 2023-10-30T12:32:28.671 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9832 ms) 2023-10-30T12:32:28.671 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2023-10-30T12:32:28.671 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698669052,0 2023-10-30T12:32:28.671 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: first is 1698669052 2023-10-30T12:32:28.672 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698669052,0 2023-10-30T12:32:28.672 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698669052,0 2023-10-30T12:32:28.672 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698669052,1698669054,1698669055,0 2023-10-30T12:32:28.672 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698669052,1698669054,1698669055,0 2023-10-30T12:32:28.673 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698669052,1698669054,1698669055,0 2023-10-30T12:32:28.673 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698669052,1698669054,1698669055,1698669057,1698669058,0 2023-10-30T12:32:28.673 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698669052,1698669054,1698669055,1698669057,1698669058,0 2023-10-30T12:32:28.673 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698669052,1698669054,1698669055,1698669057,1698669058,0 2023-10-30T12:32:28.673 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698669052,1698669054,1698669055,1698669057,1698669058,1698669060,1698669061,0 2023-10-30T12:32:28.673 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698669052,1698669054,1698669055,1698669057,1698669058,1698669060,1698669061,0 2023-10-30T12:32:28.674 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698669052,1698669054,1698669055,1698669057,1698669058,1698669060,1698669061,0 2023-10-30T12:32:28.674 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: got ls 1698669055,1698669057,1698669058,1698669060,1698669061,1698669063,1698669064,0 2023-10-30T12:32:28.674 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: first now 1698669055, trimmed 2023-10-30T12:32:28.675 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22232 ms) 2023-10-30T12:32:28.675 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2023-10-30T12:32:28.675 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: foo0 2023-10-30T12:32:28.675 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-10-30T12:32:28.675 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (16169 ms) 2023-10-30T12:32:28.676 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2023-10-30T12:32:28.676 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (20314 ms) 2023-10-30T12:32:28.676 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2023-10-30T12:32:28.677 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (23845 ms) 2023-10-30T12:32:28.677 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2023-10-30T12:32:28.677 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5591 ms) 2023-10-30T12:32:28.678 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2023-10-30T12:32:28.678 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4506 ms) 2023-10-30T12:32:28.678 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2023-10-30T12:32:28.679 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4542 ms) 2023-10-30T12:32:28.679 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2023-10-30T12:32:28.679 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4479 ms) 2023-10-30T12:32:28.679 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (265117 ms total) 2023-10-30T12:32:28.680 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: 2023-10-30T12:32:28.680 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [----------] Global test environment tear-down 2023-10-30T12:32:28.680 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1247336 ms total) 2023-10-30T12:32:28.681 INFO:tasks.workunit.client.0.smithi003.stdout: api_tier_pp: [ PASSED ] 77 tests. 2023-10-30T12:32:28.684 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.684 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91849 2023-10-30T12:32:28.684 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 91849 2023-10-30T12:32:28.685 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.685 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92058 2023-10-30T12:32:28.685 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92058 2023-10-30T12:32:28.685 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.685 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92572 2023-10-30T12:32:28.686 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92572 2023-10-30T12:32:28.686 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.686 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92300 2023-10-30T12:32:28.686 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92300 2023-10-30T12:32:28.686 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.687 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92158 2023-10-30T12:32:28.687 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92158 2023-10-30T12:32:28.687 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.687 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91926 2023-10-30T12:32:28.688 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 91926 2023-10-30T12:32:28.688 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.688 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92236 2023-10-30T12:32:28.688 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92236 2023-10-30T12:32:28.688 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.688 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91970 2023-10-30T12:32:28.689 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 91970 2023-10-30T12:32:28.689 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.689 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92470 2023-10-30T12:32:28.689 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92470 2023-10-30T12:32:28.689 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.690 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91824 2023-10-30T12:32:28.690 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 91824 2023-10-30T12:32:28.690 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.690 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91886 2023-10-30T12:32:28.690 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 91886 2023-10-30T12:32:28.691 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.691 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92515 2023-10-30T12:32:28.691 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92515 2023-10-30T12:32:28.691 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.691 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91815 2023-10-30T12:32:28.692 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 91815 2023-10-30T12:32:28.692 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.692 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91836 2023-10-30T12:32:28.692 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 91836 2023-10-30T12:32:28.692 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.693 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92081 2023-10-30T12:32:28.693 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92081 2023-10-30T12:32:28.693 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.693 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92208 2023-10-30T12:32:28.693 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92208 2023-10-30T12:32:28.694 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.694 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92122 2023-10-30T12:32:28.694 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92122 2023-10-30T12:32:28.694 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.695 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92529 2023-10-30T12:32:28.695 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92529 2023-10-30T12:32:28.695 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.695 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92440 2023-10-30T12:32:28.695 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92440 2023-10-30T12:32:28.696 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.696 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92028 2023-10-30T12:32:28.696 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92028 2023-10-30T12:32:28.697 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.697 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92341 2023-10-30T12:32:28.697 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92341 2023-10-30T12:32:28.698 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.698 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91808 2023-10-30T12:32:28.699 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 91808 2023-10-30T12:32:28.699 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.699 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91866 2023-10-30T12:32:28.699 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 91866 2023-10-30T12:32:28.700 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.700 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92258 2023-10-30T12:32:28.700 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92258 2023-10-30T12:32:28.701 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.701 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=91905 2023-10-30T12:32:28.701 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 91905 2023-10-30T12:32:28.701 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.702 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92371 2023-10-30T12:32:28.702 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92371 2023-10-30T12:32:28.702 INFO:tasks.workunit.client.0.smithi003.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:32:28.703 INFO:tasks.workunit.client.0.smithi003.stderr:+ pid=92410 2023-10-30T12:32:28.703 INFO:tasks.workunit.client.0.smithi003.stderr:+ wait 92410 2023-10-30T12:32:28.703 INFO:tasks.workunit.client.0.smithi003.stderr:+ exit 0 2023-10-30T12:32:28.703 INFO:tasks.workunit.client.0.smithi003.stderr:+ cleanup 2023-10-30T12:32:28.703 INFO:tasks.workunit.client.0.smithi003.stderr:+ pkill -P 91803 2023-10-30T12:32:28.704 INFO:tasks.workunit.client.0.smithi003.stderr:+ true 2023-10-30T12:32:28.705 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:32:28.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:32:28.739 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2023-10-30T12:32:28.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-30T12:32:28.807 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:28.716 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.808 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:28.717 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.817 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:28.725 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.878 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:28.814 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.879 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:28.815 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.891 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:28.827 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.892 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:28.828 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.893 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:28.829 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:28.904 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:28.840 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:29.073 DEBUG:teuthology.parallel:result is None 2023-10-30T12:32:29.073 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-10-30T12:32:29.109 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-10-30T12:32:29.110 DEBUG:teuthology.orchestra.run.smithi003:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-10-30T12:32:29.286 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-10-30T12:32:29.286 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-30T12:32:29.295 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T12:32:29.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:32:29.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.404+0000 7f1077cbb640 10 monclient: get_monmap_and_config 2023-10-30T12:32:29.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.404+0000 7f1077cbb640 10 monclient: build_initial_monmap 2023-10-30T12:32:29.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.404+0000 7f1077cbb640 10 monclient: monmap: 2023-10-30T12:32:29.407 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:29.407 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:29.407 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:29.405887+0000 2023-10-30T12:32:29.407 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:29.405887+0000 2023-10-30T12:32:29.407 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:29.408 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:29.408 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:29.408 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:29.408 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:29.408 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:29.409 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:29.409 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:29.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.405+0000 7f1077cbb640 1 Processor -- start 2023-10-30T12:32:29.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.405+0000 7f1077cbb640 1 -- start start 2023-10-30T12:32:29.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.405+0000 7f1077cbb640 10 monclient: init 2023-10-30T12:32:29.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.406+0000 7f1077cbb640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:29.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.406+0000 7f1077cbb640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2023-10-30T12:32:29.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.406+0000 7f1077cbb640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f10700b4390 0x7f107013ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:29.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.406+0000 7f1077cbb640 10 monclient(hunting): picked mon.noname-a con 0x7f10700b4390 addr v2:172.21.15.3:3300/0 2023-10-30T12:32:29.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.406+0000 7f1077cbb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10700b12d0 addr v1:172.21.15.3:6789/0 2023-10-30T12:32:29.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.406+0000 7f1077cbb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1070141910 addr v1:172.21.15.92:6789/0 2023-10-30T12:32:29.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.406+0000 7f1077cbb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10701424b0 con 0x7f10700b12d0 2023-10-30T12:32:29.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.406+0000 7f1077cbb640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10700aa440 con 0x7f1070141910 2023-10-30T12:32:29.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.406+0000 7f1077cbb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:29.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.406+0000 7f1077cbb640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1070067a10 con 0x7f10700b4390 2023-10-30T12:32:29.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.406+0000 7f1077cbb640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:29.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.406+0000 7f1077cbb640 10 monclient(hunting): authenticate will time out at 2740.620452s 2023-10-30T12:32:29.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.407+0000 7f1076258640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f1070141910 0x7f1070141da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.3:59580/0 (socket says 172.21.15.3:59580) 2023-10-30T12:32:29.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.407+0000 7f1076258640 1 -- 172.21.15.3:0/2289720136 learned_addr learned my addr 172.21.15.3:0/2289720136 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:32:29.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.409+0000 7f1074a55640 1 -- 172.21.15.3:0/2289720136 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501535602 0 0) 0x7f1068001490 con 0x7f1070141910 2023-10-30T12:32:29.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.409+0000 7f1074a55640 10 monclient(hunting): _negotiate 2023-10-30T12:32:29.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.409+0000 7f1074a55640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:29.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.409+0000 7f1074a55640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:29.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.409+0000 7f1074a55640 10 monclient(hunting): my global_id is 21784 2023-10-30T12:32:29.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.409+0000 7f1074a55640 1 -- 172.21.15.3:0/2289720136 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f105c003520 con 0x7f1070141910 2023-10-30T12:32:29.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 1 -- 172.21.15.3:0/2289720136 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 55318314 0 0) 0x7f1068001890 con 0x7f1070141910 2023-10-30T12:32:29.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 1 -- 172.21.15.3:0/2289720136 >> v2:172.21.15.3:3300/0 conn(0x7f10700b4390 msgr2=0x7f107013ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:29.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 1 --2- 172.21.15.3:0/2289720136 >> v2:172.21.15.3:3300/0 conn(0x7f10700b4390 0x7f107013ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 1 -- 172.21.15.3:0/2289720136 >> v1:172.21.15.3:6789/0 conn(0x7f10700b12d0 legacy=0x7f107013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:29.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient: _finish_hunting 0 2023-10-30T12:32:29.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:32:29.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:29.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.92:6789/0 2023-10-30T12:32:29.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 1 -- 172.21.15.3:0/2289720136 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10700a8800 con 0x7f1070141910 2023-10-30T12:32:29.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient: _finish_auth 0 2023-10-30T12:32:29.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient: _check_auth_tickets 2023-10-30T12:32:29.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:29.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1077cbb640 5 monclient: authenticate success, global_id 21784 2023-10-30T12:32:29.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 1 -- 172.21.15.3:0/2289720136 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1777801622 0 0) 0x7f1068001d30 con 0x7f1070141910 2023-10-30T12:32:29.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:29.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:32:29.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient: dump: 2023-10-30T12:32:29.419 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:29.420 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:29.420 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:29.420 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:29.420 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:29.420 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:29.421 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:29.421 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:29.421 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:29.421 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:29.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] but i'm connected to v1:172.21.15.92:6789/0, reconnecting 2023-10-30T12:32:29.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:29.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 1 -- 172.21.15.3:0/2289720136 >> v1:172.21.15.92:6789/0 conn(0x7f1070141910 legacy=0x7f1070141da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:29.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:32:29.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 1 --2- 172.21.15.3:0/2289720136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f105c003970 0x7f105c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:29.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient(hunting): picked mon.a con 0x7f105c003970 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:29.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 1 --2- 172.21.15.3:0/2289720136 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f105c0047c0 0x7f105c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:29.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient(hunting): picked mon.c con 0x7f105c0047c0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:29.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 1 --2- 172.21.15.3:0/2289720136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f105c0052a0 0x7f105c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:29.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient(hunting): picked mon.b con 0x7f105c0052a0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:29.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:29.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 1 -- 172.21.15.3:0/2289720136 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1070067a10 con 0x7f105c003970 2023-10-30T12:32:29.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:29.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 1 -- 172.21.15.3:0/2289720136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105c00c190 con 0x7f105c0052a0 2023-10-30T12:32:29.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:29.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 1 -- 172.21.15.3:0/2289720136 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105c00c360 con 0x7f105c0047c0 2023-10-30T12:32:29.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:29.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1074a55640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:32:29.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.413+0000 7f1077cbb640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:32:29.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.416+0000 7f1076258640 1 --2- 172.21.15.3:0/2289720136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f105c0052a0 0x7f105c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:29.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.416+0000 7f1076258640 10 monclient(hunting): get_auth_request con 0x7f105c0052a0 auth_method 0 2023-10-30T12:32:29.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.416+0000 7f1076258640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:29.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.416+0000 7f1076258640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:29.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.416+0000 7f1076258640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:32:29.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.417+0000 7f1076258640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:29.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.417+0000 7f1076258640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:29.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.417+0000 7f1076258640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:32:29.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.419+0000 7f1076258640 10 monclient(hunting): handle_auth_done global_id 21784 payload 293 2023-10-30T12:32:29.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.419+0000 7f1076258640 1 -- 172.21.15.3:0/2289720136 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f105c0047c0 msgr2=0x7f105c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:29.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.419+0000 7f1076258640 1 --2- 172.21.15.3:0/2289720136 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f105c0047c0 0x7f105c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.419+0000 7f1076258640 1 -- 172.21.15.3:0/2289720136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f105c003970 msgr2=0x7f105c003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:29.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.419+0000 7f1076258640 1 --2- 172.21.15.3:0/2289720136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f105c003970 0x7f105c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.419+0000 7f1076258640 10 monclient: _finish_hunting 0 2023-10-30T12:32:29.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.419+0000 7f1076258640 1 monclient: found mon.b 2023-10-30T12:32:29.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.419+0000 7f1076258640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:29.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.419+0000 7f1076258640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:29.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.419+0000 7f1076258640 1 -- 172.21.15.3:0/2289720136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1068001890 con 0x7f105c0052a0 2023-10-30T12:32:29.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.419+0000 7f1076258640 10 monclient: _finish_auth 0 2023-10-30T12:32:29.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.419+0000 7f1076258640 10 monclient: _check_auth_tickets 2023-10-30T12:32:29.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.419+0000 7f1076258640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:29.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.419+0000 7f1076258640 1 -- 172.21.15.3:0/2289720136 reap_dead start 2023-10-30T12:32:29.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.421+0000 7f1076258640 1 --2- 172.21.15.3:0/2289720136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f105c0052a0 0x7f105c00b980 secure :-1 s=READY pgs=5296 cs=0 l=1 rev1=1 crypto rx=0x7f1068002230 tx=0x7f105c0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c4e7d6f78e20812 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:29.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.424+0000 7f1074a55640 1 -- 172.21.15.3:0/2289720136 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10680072d0 con 0x7f105c0052a0 2023-10-30T12:32:29.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.424+0000 7f1074a55640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:29.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.424+0000 7f1074a55640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:29.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.424+0000 7f1074a55640 10 monclient: dump: 2023-10-30T12:32:29.433 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:29.433 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:29.433 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:29.433 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:29.433 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:29.434 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:29.434 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:29.434 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:29.434 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:29.434 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:29.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.424+0000 7f1074a55640 1 -- 172.21.15.3:0/2289720136 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1068007460 con 0x7f105c0052a0 2023-10-30T12:32:29.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.424+0000 7f1077cbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:32:29.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.424+0000 7f1074a55640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:29.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.424+0000 7f1074a55640 1 -- 172.21.15.3:0/2289720136 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1068007820 con 0x7f105c0052a0 2023-10-30T12:32:29.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.424+0000 7f1077cbb640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:29.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.424+0000 7f1077cbb640 4 set_mon_vals no callback set 2023-10-30T12:32:29.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.424+0000 7f1074a55640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:29.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.424+0000 7f1074a55640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:29.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.424+0000 7f1074a55640 10 monclient: dump: 2023-10-30T12:32:29.436 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:29.436 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:29.437 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:29.437 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:29.437 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:29.437 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:29.437 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:29.438 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:29.438 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:29.438 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:29.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.425+0000 7f1077cbb640 10 monclient: shutdown 2023-10-30T12:32:29.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.425+0000 7f1077cbb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.425+0000 7f1077cbb640 1 -- 172.21.15.3:0/2289720136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f105c0052a0 msgr2=0x7f105c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.425+0000 7f1077cbb640 1 --2- 172.21.15.3:0/2289720136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f105c0052a0 0x7f105c00b980 secure :-1 s=READY pgs=5296 cs=0 l=1 rev1=1 crypto rx=0x7f1068002230 tx=0x7f105c0057f0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.425+0000 7f1077cbb640 1 -- 172.21.15.3:0/2289720136 shutdown_connections 2023-10-30T12:32:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.425+0000 7f1077cbb640 1 --2- 172.21.15.3:0/2289720136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f105c0052a0 0x7f105c00b980 unknown :-1 s=CLOSED pgs=5296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.425+0000 7f1077cbb640 1 -- 172.21.15.3:0/2289720136 >> 172.21.15.3:0/2289720136 conn(0x7f1070065ed0 msgr2=0x7f10700662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.425+0000 7f1077cbb640 1 -- 172.21.15.3:0/2289720136 shutdown_connections 2023-10-30T12:32:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.425+0000 7f1077cbb640 1 -- 172.21.15.3:0/2289720136 wait complete. 2023-10-30T12:32:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.426+0000 7f1077cbb640 10 monclient: build_initial_monmap 2023-10-30T12:32:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.426+0000 7f1077cbb640 10 monclient: monmap: 2023-10-30T12:32:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:29.442 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:29.442 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:29.442 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:29.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.426+0000 7f1077cbb640 1 librados: starting msgr at 2023-10-30T12:32:29.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.426+0000 7f1077cbb640 1 librados: starting objecter 2023-10-30T12:32:29.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.426+0000 7f1077cbb640 1 Processor -- start 2023-10-30T12:32:29.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.426+0000 7f1077cbb640 1 -- start start 2023-10-30T12:32:29.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.426+0000 7f1077cbb640 1 librados: setting wanted keys 2023-10-30T12:32:29.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.426+0000 7f1077cbb640 1 librados: calling monclient init 2023-10-30T12:32:29.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.426+0000 7f1077cbb640 10 monclient: init 2023-10-30T12:32:29.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:29.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:32:29.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f105c0052a0 0x7f10701cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:29.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 10 monclient(hunting): picked mon.noname-a con 0x7f105c0052a0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:29.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1070141530 0x7f1070139f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:29.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 10 monclient(hunting): picked mon.noname-c con 0x7f1070141530 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:29.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f107013a6d0 0x7f10701d6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:29.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 10 monclient(hunting): picked mon.noname-b con 0x7f107013a6d0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:29.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:29.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10700b2d90 con 0x7f105c0052a0 2023-10-30T12:32:29.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:29.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107013d8f0 con 0x7f107013a6d0 2023-10-30T12:32:29.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:29.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107013a4b0 con 0x7f1070141530 2023-10-30T12:32:29.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:29.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.427+0000 7f1077cbb640 10 monclient(hunting): authenticate will time out at 2740.641489s 2023-10-30T12:32:29.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.428+0000 7f1076258640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f107013a6d0 0x7f10701d6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:29.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.428+0000 7f1076258640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f107013a6d0 0x7f10701d6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:33198/0 (socket says 172.21.15.3:33198) 2023-10-30T12:32:29.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.428+0000 7f1076258640 1 -- 172.21.15.3:0/3885867467 learned_addr learned my addr 172.21.15.3:0/3885867467 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:29.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.428+0000 7f1076258640 10 monclient(hunting): get_auth_request con 0x7f107013a6d0 auth_method 0 2023-10-30T12:32:29.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.428+0000 7f1076258640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:29.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.428+0000 7f1076258640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:29.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.428+0000 7f1076258640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:29.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.429+0000 7f1076258640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:29.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.429+0000 7f1076258640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:29.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.429+0000 7f1076258640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:29.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.432+0000 7f1076258640 10 monclient(hunting): handle_auth_done global_id 21787 payload 824 2023-10-30T12:32:29.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.432+0000 7f1076258640 1 -- 172.21.15.3:0/3885867467 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1070141530 msgr2=0x7f1070139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:29.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.432+0000 7f1076258640 1 --2- 172.21.15.3:0/3885867467 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1070141530 0x7f1070139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.432+0000 7f1076258640 1 -- 172.21.15.3:0/3885867467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f105c0052a0 msgr2=0x7f10701cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:29.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.432+0000 7f1076258640 1 --2- 172.21.15.3:0/3885867467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f105c0052a0 0x7f10701cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.432+0000 7f1076258640 10 monclient: _finish_hunting 0 2023-10-30T12:32:29.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.432+0000 7f1076258640 1 monclient: found mon.noname-b 2023-10-30T12:32:29.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.432+0000 7f1076258640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:29.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.432+0000 7f1076258640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:29.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.432+0000 7f1076258640 1 -- 172.21.15.3:0/3885867467 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10701d6a70 con 0x7f107013a6d0 2023-10-30T12:32:29.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.434+0000 7f1076258640 1 --2- 172.21.15.3:0/3885867467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f107013a6d0 0x7f10701d6530 secure :-1 s=READY pgs=5297 cs=0 l=1 rev1=1 crypto rx=0x7f1068000f80 tx=0x7f10680020d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:29.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.434+0000 7f10667fc640 10 client.?.objecter ms_handle_connect 0x7f107013a6d0 2023-10-30T12:32:29.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.434+0000 7f10667fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:29.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f10667fc640 1 -- 172.21.15.3:0/3885867467 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1068001d20 con 0x7f107013a6d0 2023-10-30T12:32:29.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f10667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:29.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f10667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:29.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f10667fc640 10 monclient: dump: 2023-10-30T12:32:29.454 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:29.454 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:29.455 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:29.455 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:29.455 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:29.455 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:29.455 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:29.456 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:29.456 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:29.456 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:29.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f10667fc640 10 monclient: _finish_auth 0 2023-10-30T12:32:29.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f10667fc640 10 monclient: _check_auth_tickets 2023-10-30T12:32:29.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f10667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:29.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f10667fc640 1 -- 172.21.15.3:0/3885867467 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1068001490 con 0x7f107013a6d0 2023-10-30T12:32:29.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f1077cbb640 5 monclient: authenticate success, global_id 21787 2023-10-30T12:32:29.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f1077cbb640 10 monclient: _renew_subs 2023-10-30T12:32:29.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f1077cbb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:29.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f1077cbb640 1 -- 172.21.15.3:0/3885867467 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1070139620 con 0x7f107013a6d0 2023-10-30T12:32:29.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f10667fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:29.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f10667fc640 1 -- 172.21.15.3:0/3885867467 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10680075e0 con 0x7f107013a6d0 2023-10-30T12:32:29.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f10667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:29.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f10667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:29.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f10667fc640 10 monclient: dump: 2023-10-30T12:32:29.458 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:29.459 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:29.459 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:29.459 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:29.459 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:29.459 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:29.460 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:29.460 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:29.460 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:29.460 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:29.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f1067fff640 4 set_mon_vals no callback set 2023-10-30T12:32:29.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f1077cbb640 10 client.21787.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:29.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f1077cbb640 10 monclient: _renew_subs 2023-10-30T12:32:29.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f1077cbb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:29.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f1077cbb640 1 -- 172.21.15.3:0/3885867467 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10701d9c10 con 0x7f107013a6d0 2023-10-30T12:32:29.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.437+0000 7f1077cbb640 1 librados: init done 2023-10-30T12:32:29.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.439+0000 7f1077cbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:29.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.439+0000 7f1077cbb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:29.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.439+0000 7f1077cbb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:29.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.439+0000 7f1077cbb640 1 -- 172.21.15.3:0/3885867467 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107004e9b0 con 0x7f107013a6d0 2023-10-30T12:32:29.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.457+0000 7f10667fc640 1 -- 172.21.15.3:0/3885867467 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f1068007300 con 0x7f107013a6d0 2023-10-30T12:32:29.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.458+0000 7f10667fc640 1 --2- 172.21.15.3:0/3885867467 >> v2:172.21.15.92:6800/1750155735 conn(0x7f105406c7c0 0x7f105406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:29.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.458+0000 7f1075a57640 1 --2- 172.21.15.3:0/3885867467 >> v2:172.21.15.92:6800/1750155735 conn(0x7f105406c7c0 0x7f105406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:29.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.458+0000 7f1075a57640 10 monclient: get_auth_request con 0x7f105406c7c0 auth_method 0 2023-10-30T12:32:29.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.458+0000 7f1075a57640 1 --2- 172.21.15.3:0/3885867467 >> v2:172.21.15.92:6800/1750155735 conn(0x7f105406c7c0 0x7f105406ec80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f106c009770 tx=0x7f106c006d70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:29.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.458+0000 7f10667fc640 10 client.21787.objecter ms_handle_connect 0x7f105406c7c0 2023-10-30T12:32:29.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.684+0000 7f10667fc640 1 -- 172.21.15.3:0/3885867467 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(899..899 src has 627..899) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f106807a260 con 0x7f107013a6d0 2023-10-30T12:32:29.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.684+0000 7f10667fc640 10 client.21787.objecter ms_dispatch 0x7f1070000b90 osd_map(899..899 src has 627..899) v4 2023-10-30T12:32:29.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.684+0000 7f10667fc640 3 client.21787.objecter handle_osd_map got epochs [899,899] > 0 2023-10-30T12:32:29.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.684+0000 7f10667fc640 3 client.21787.objecter handle_osd_map decoding full epoch 899 2023-10-30T12:32:29.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.684+0000 7f10667fc640 20 client.21787.objecter dump_active .. 0 homeless 2023-10-30T12:32:29.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.693+0000 7f10667fc640 1 -- 172.21.15.3:0/3885867467 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f106800d040 con 0x7f107013a6d0 2023-10-30T12:32:29.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.693+0000 7f10667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:29.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.693+0000 7f10667fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:32:29.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.877+0000 7f1077cbb640 1 -- 172.21.15.3:0/3885867467 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f10700b4770 con 0x7f105406c7c0 2023-10-30T12:32:29.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.882+0000 7f10667fc640 1 -- 172.21.15.3:0/3885867467 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151652 (secure 0 0 0) 0x7f106c00b4b0 con 0x7f105406c7c0 2023-10-30T12:32:29.885 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:32:29.886 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-10-30T12:32:29.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 10 librados: watch_flush enter 2023-10-30T12:32:29.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 10 librados: watch_flush exit 2023-10-30T12:32:29.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 20 client.21787.objecter shutdown clearing up homeless session... 2023-10-30T12:32:29.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 10 client.21787.objecter successfully canceled tick 2023-10-30T12:32:29.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 1 -- 172.21.15.3:0/3885867467 >> v2:172.21.15.92:6800/1750155735 conn(0x7f105406c7c0 msgr2=0x7f105406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:29.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 1 --2- 172.21.15.3:0/3885867467 >> v2:172.21.15.92:6800/1750155735 conn(0x7f105406c7c0 0x7f105406ec80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f106c009770 tx=0x7f106c006d70 comp rx=0 tx=0).stop 2023-10-30T12:32:29.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 10 monclient: shutdown 2023-10-30T12:32:29.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:29.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 1 -- 172.21.15.3:0/3885867467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f107013a6d0 msgr2=0x7f10701d6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:29.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 1 --2- 172.21.15.3:0/3885867467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f107013a6d0 0x7f10701d6530 secure :-1 s=READY pgs=5297 cs=0 l=1 rev1=1 crypto rx=0x7f1068000f80 tx=0x7f10680020d0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 1 -- 172.21.15.3:0/3885867467 shutdown_connections 2023-10-30T12:32:29.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 1 --2- 172.21.15.3:0/3885867467 >> v2:172.21.15.92:6800/1750155735 conn(0x7f105406c7c0 0x7f105406ec80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 1 --2- 172.21.15.3:0/3885867467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f107013a6d0 0x7f10701d6530 unknown :-1 s=CLOSED pgs=5297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 1 --2- 172.21.15.3:0/3885867467 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1070141530 0x7f1070139f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 1 --2- 172.21.15.3:0/3885867467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f105c0052a0 0x7f10701cb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.884+0000 7f1077cbb640 1 -- 172.21.15.3:0/3885867467 >> 172.21.15.3:0/3885867467 conn(0x7f1070065ed0 msgr2=0x7f1070138540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:29.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.885+0000 7f1077cbb640 1 -- 172.21.15.3:0/3885867467 shutdown_connections 2023-10-30T12:32:29.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.885+0000 7f1077cbb640 1 -- 172.21.15.3:0/3885867467 wait complete. 2023-10-30T12:32:29.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:29.885+0000 7f1077cbb640 1 librados: shutdown 2023-10-30T12:32:29.897 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1576,"stamp":"2023-10-30T12:32:28.556188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":581440,"kb_used_data":55772,"kb_used_omap":12,"kb_used_meta":525491,"kb_avail":754393280,"statfs":{"total":773094113280,"available":772498718720,"internally_reserved":0,"allocated":57110528,"data_stored":28189101,"data_compressed":5094,"data_compressed_allocated":57344,"data_compressed_original":909312,"omap_allocated":12743,"internal_metadata":538103353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":427,"apply_latency_ms":427,"commit_latency_ns":427000000,"apply_latency_ns":427000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.451149"},"pg_stats":[{"pgid":"24.4","version":"0'0","reported_seq":849,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.695695+0000","last_change":"2023-10-30T12:11:50.816793+0000","last_active":"2023-10-30T12:32:22.695695+0000","last_peered":"2023-10-30T12:32:22.695695+0000","last_clean":"2023-10-30T12:32:22.695695+0000","last_became_active":"2023-10-30T12:11:50.811294+0000","last_became_peered":"2023-10-30T12:11:50.811294+0000","last_unstale":"2023-10-30T12:32:22.695695+0000","last_undegraded":"2023-10-30T12:32:22.695695+0000","last_fullsized":"2023-10-30T12:32:22.695695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:34:55.305107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":878,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.776547+0000","last_change":"2023-10-30T12:11:58.574669+0000","last_active":"2023-10-30T12:32:22.776547+0000","last_peered":"2023-10-30T12:32:22.776547+0000","last_clean":"2023-10-30T12:32:22.776547+0000","last_became_active":"2023-10-30T12:11:58.558569+0000","last_became_peered":"2023-10-30T12:11:58.558569+0000","last_unstale":"2023-10-30T12:32:22.776547+0000","last_undegraded":"2023-10-30T12:32:22.776547+0000","last_fullsized":"2023-10-30T12:32:22.776547+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:17:50.972635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":878,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.796700+0000","last_change":"2023-10-30T12:11:58.500558+0000","last_active":"2023-10-30T12:32:22.796700+0000","last_peered":"2023-10-30T12:32:22.796700+0000","last_clean":"2023-10-30T12:32:22.796700+0000","last_became_active":"2023-10-30T12:11:58.495252+0000","last_became_peered":"2023-10-30T12:11:58.495252+0000","last_unstale":"2023-10-30T12:32:22.796700+0000","last_undegraded":"2023-10-30T12:32:22.796700+0000","last_fullsized":"2023-10-30T12:32:22.796700+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:27.257928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":863,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.877568+0000","last_change":"2023-10-30T12:12:25.951506+0000","last_active":"2023-10-30T12:32:22.877568+0000","last_peered":"2023-10-30T12:32:22.877568+0000","last_clean":"2023-10-30T12:32:22.877568+0000","last_became_active":"2023-10-30T12:12:25.941872+0000","last_became_peered":"2023-10-30T12:12:25.941872+0000","last_unstale":"2023-10-30T12:32:22.877568+0000","last_undegraded":"2023-10-30T12:32:22.877568+0000","last_fullsized":"2023-10-30T12:32:22.877568+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:34.170866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":868,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.118078+0000","last_change":"2023-10-30T12:11:51.588272+0000","last_active":"2023-10-30T12:32:26.118078+0000","last_peered":"2023-10-30T12:32:26.118078+0000","last_clean":"2023-10-30T12:32:26.118078+0000","last_became_active":"2023-10-30T12:11:51.581699+0000","last_became_peered":"2023-10-30T12:11:51.581699+0000","last_unstale":"2023-10-30T12:32:26.118078+0000","last_undegraded":"2023-10-30T12:32:26.118078+0000","last_fullsized":"2023-10-30T12:32:26.118078+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:08.811631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1","version":"50'7","reported_seq":862,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:32:23.390178+0000","last_change":"2023-10-30T12:12:02.329571+0000","last_active":"2023-10-30T12:32:23.390178+0000","last_peered":"2023-10-30T12:32:23.390178+0000","last_clean":"2023-10-30T12:32:23.390178+0000","last_became_active":"2023-10-30T12:12:02.320185+0000","last_became_peered":"2023-10-30T12:12:02.320185+0000","last_unstale":"2023-10-30T12:32:23.390178+0000","last_undegraded":"2023-10-30T12:32:23.390178+0000","last_fullsized":"2023-10-30T12:32:23.390178+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:19:24.782158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":849,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:32:23.386484+0000","last_change":"2023-10-30T12:12:02.278839+0000","last_active":"2023-10-30T12:32:23.386484+0000","last_peered":"2023-10-30T12:32:23.386484+0000","last_clean":"2023-10-30T12:32:23.386484+0000","last_became_active":"2023-10-30T12:12:02.269307+0000","last_became_peered":"2023-10-30T12:12:02.269307+0000","last_unstale":"2023-10-30T12:32:23.386484+0000","last_undegraded":"2023-10-30T12:32:23.386484+0000","last_fullsized":"2023-10-30T12:32:23.386484+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:34.516157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":849,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:32:23.400939+0000","last_change":"2023-10-30T12:11:55.586707+0000","last_active":"2023-10-30T12:32:23.400939+0000","last_peered":"2023-10-30T12:32:23.400939+0000","last_clean":"2023-10-30T12:32:23.400939+0000","last_became_active":"2023-10-30T12:11:55.581296+0000","last_became_peered":"2023-10-30T12:11:55.581296+0000","last_unstale":"2023-10-30T12:32:23.400939+0000","last_undegraded":"2023-10-30T12:32:23.400939+0000","last_fullsized":"2023-10-30T12:32:23.400939+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:04.280909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":863,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.793266+0000","last_change":"2023-10-30T12:12:25.845763+0000","last_active":"2023-10-30T12:32:22.793266+0000","last_peered":"2023-10-30T12:32:22.793266+0000","last_clean":"2023-10-30T12:32:22.793266+0000","last_became_active":"2023-10-30T12:12:25.833774+0000","last_became_peered":"2023-10-30T12:12:25.833774+0000","last_unstale":"2023-10-30T12:32:22.793266+0000","last_undegraded":"2023-10-30T12:32:22.793266+0000","last_fullsized":"2023-10-30T12:32:22.793266+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:17.229351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":863,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.834630+0000","last_change":"2023-10-30T12:12:09.181066+0000","last_active":"2023-10-30T12:32:22.834630+0000","last_peered":"2023-10-30T12:32:22.834630+0000","last_clean":"2023-10-30T12:32:22.834630+0000","last_became_active":"2023-10-30T12:12:09.094001+0000","last_became_peered":"2023-10-30T12:12:09.094001+0000","last_unstale":"2023-10-30T12:32:22.834630+0000","last_undegraded":"2023-10-30T12:32:22.834630+0000","last_fullsized":"2023-10-30T12:32:22.834630+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:36.491842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":867,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.192648+0000","last_change":"2023-10-30T12:12:04.088614+0000","last_active":"2023-10-30T12:32:26.192648+0000","last_peered":"2023-10-30T12:32:26.192648+0000","last_clean":"2023-10-30T12:32:26.192648+0000","last_became_active":"2023-10-30T12:12:04.077824+0000","last_became_peered":"2023-10-30T12:12:04.077824+0000","last_unstale":"2023-10-30T12:32:26.192648+0000","last_undegraded":"2023-10-30T12:32:26.192648+0000","last_fullsized":"2023-10-30T12:32:26.192648+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:34.462067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":847,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.735641+0000","last_change":"2023-10-30T12:11:50.993004+0000","last_active":"2023-10-30T12:32:22.735641+0000","last_peered":"2023-10-30T12:32:22.735641+0000","last_clean":"2023-10-30T12:32:22.735641+0000","last_became_active":"2023-10-30T12:11:50.987299+0000","last_became_peered":"2023-10-30T12:11:50.987299+0000","last_unstale":"2023-10-30T12:32:22.735641+0000","last_undegraded":"2023-10-30T12:32:22.735641+0000","last_fullsized":"2023-10-30T12:32:22.735641+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:02:34.008030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":848,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.716192+0000","last_change":"2023-10-30T12:11:51.028381+0000","last_active":"2023-10-30T12:32:22.716192+0000","last_peered":"2023-10-30T12:32:22.716192+0000","last_clean":"2023-10-30T12:32:22.716192+0000","last_became_active":"2023-10-30T12:11:51.023075+0000","last_became_peered":"2023-10-30T12:11:51.023075+0000","last_unstale":"2023-10-30T12:32:22.716192+0000","last_undegraded":"2023-10-30T12:32:22.716192+0000","last_fullsized":"2023-10-30T12:32:22.716192+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:23:10.299251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":870,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.874740+0000","last_change":"2023-10-30T12:10:17.745135+0000","last_active":"2023-10-30T12:32:22.874740+0000","last_peered":"2023-10-30T12:32:22.874740+0000","last_clean":"2023-10-30T12:32:22.874740+0000","last_became_active":"2023-10-30T12:10:17.738241+0000","last_became_peered":"2023-10-30T12:10:17.738241+0000","last_unstale":"2023-10-30T12:32:22.874740+0000","last_undegraded":"2023-10-30T12:32:22.874740+0000","last_fullsized":"2023-10-30T12:32:22.874740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:01.788933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":857,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.835701+0000","last_change":"2023-10-30T12:11:52.264262+0000","last_active":"2023-10-30T12:32:22.835701+0000","last_peered":"2023-10-30T12:32:22.835701+0000","last_clean":"2023-10-30T12:32:22.835701+0000","last_became_active":"2023-10-30T12:11:52.257573+0000","last_became_peered":"2023-10-30T12:11:52.257573+0000","last_unstale":"2023-10-30T12:32:22.835701+0000","last_undegraded":"2023-10-30T12:32:22.835701+0000","last_fullsized":"2023-10-30T12:32:22.835701+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:48:44.255249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":874,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.195697+0000","last_change":"2023-10-30T12:10:17.790588+0000","last_active":"2023-10-30T12:32:26.195697+0000","last_peered":"2023-10-30T12:32:26.195697+0000","last_clean":"2023-10-30T12:32:26.195697+0000","last_became_active":"2023-10-30T12:10:17.763227+0000","last_became_peered":"2023-10-30T12:10:17.763227+0000","last_unstale":"2023-10-30T12:32:26.195697+0000","last_undegraded":"2023-10-30T12:32:26.195697+0000","last_fullsized":"2023-10-30T12:32:26.195697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:40:26.455087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":878,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.818285+0000","last_change":"2023-10-30T12:11:55.621845+0000","last_active":"2023-10-30T12:32:22.818285+0000","last_peered":"2023-10-30T12:32:22.818285+0000","last_clean":"2023-10-30T12:32:22.818285+0000","last_became_active":"2023-10-30T12:11:55.613173+0000","last_became_peered":"2023-10-30T12:11:55.613173+0000","last_unstale":"2023-10-30T12:32:22.818285+0000","last_undegraded":"2023-10-30T12:32:22.818285+0000","last_fullsized":"2023-10-30T12:32:22.818285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:41.422061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":870,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.810818+0000","last_change":"2023-10-30T12:10:16.912669+0000","last_active":"2023-10-30T12:32:22.810818+0000","last_peered":"2023-10-30T12:32:22.810818+0000","last_clean":"2023-10-30T12:32:22.810818+0000","last_became_active":"2023-10-30T12:10:16.907205+0000","last_became_peered":"2023-10-30T12:10:16.907205+0000","last_unstale":"2023-10-30T12:32:22.810818+0000","last_undegraded":"2023-10-30T12:32:22.810818+0000","last_fullsized":"2023-10-30T12:32:22.810818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:48.591409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":858,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.913561+0000","last_change":"2023-10-30T12:11:53.046581+0000","last_active":"2023-10-30T12:32:22.913561+0000","last_peered":"2023-10-30T12:32:22.913561+0000","last_clean":"2023-10-30T12:32:22.913561+0000","last_became_active":"2023-10-30T12:11:52.909795+0000","last_became_peered":"2023-10-30T12:11:52.909795+0000","last_unstale":"2023-10-30T12:32:22.913561+0000","last_undegraded":"2023-10-30T12:32:22.913561+0000","last_fullsized":"2023-10-30T12:32:22.913561+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:40.643512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":885,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.866498+0000","last_change":"2023-10-30T12:10:17.613569+0000","last_active":"2023-10-30T12:32:22.866498+0000","last_peered":"2023-10-30T12:32:22.866498+0000","last_clean":"2023-10-30T12:32:22.866498+0000","last_became_active":"2023-10-30T12:10:17.582475+0000","last_became_peered":"2023-10-30T12:10:17.582475+0000","last_unstale":"2023-10-30T12:32:22.866498+0000","last_undegraded":"2023-10-30T12:32:22.866498+0000","last_fullsized":"2023-10-30T12:32:22.866498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:56.738579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":856,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:32:24.549332+0000","last_change":"2023-10-30T12:11:51.597894+0000","last_active":"2023-10-30T12:32:24.549332+0000","last_peered":"2023-10-30T12:32:24.549332+0000","last_clean":"2023-10-30T12:32:24.549332+0000","last_became_active":"2023-10-30T12:11:51.581905+0000","last_became_peered":"2023-10-30T12:11:51.581905+0000","last_unstale":"2023-10-30T12:32:24.549332+0000","last_undegraded":"2023-10-30T12:32:24.549332+0000","last_fullsized":"2023-10-30T12:32:24.549332+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:04.401220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":870,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.787493+0000","last_change":"2023-10-30T12:10:17.562627+0000","last_active":"2023-10-30T12:32:22.787493+0000","last_peered":"2023-10-30T12:32:22.787493+0000","last_clean":"2023-10-30T12:32:22.787493+0000","last_became_active":"2023-10-30T12:10:17.553523+0000","last_became_peered":"2023-10-30T12:10:17.553523+0000","last_unstale":"2023-10-30T12:32:22.787493+0000","last_undegraded":"2023-10-30T12:32:22.787493+0000","last_fullsized":"2023-10-30T12:32:22.787493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:05:16.522019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":848,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.701459+0000","last_change":"2023-10-30T12:11:50.971305+0000","last_active":"2023-10-30T12:32:22.701459+0000","last_peered":"2023-10-30T12:32:22.701459+0000","last_clean":"2023-10-30T12:32:22.701459+0000","last_became_active":"2023-10-30T12:11:50.965758+0000","last_became_peered":"2023-10-30T12:11:50.965758+0000","last_unstale":"2023-10-30T12:32:22.701459+0000","last_undegraded":"2023-10-30T12:32:22.701459+0000","last_fullsized":"2023-10-30T12:32:22.701459+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:57:57.631020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":856,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.679525+0000","last_change":"2023-10-30T12:10:16.841521+0000","last_active":"2023-10-30T12:32:22.679525+0000","last_peered":"2023-10-30T12:32:22.679525+0000","last_clean":"2023-10-30T12:32:22.679525+0000","last_became_active":"2023-10-30T12:10:16.811711+0000","last_became_peered":"2023-10-30T12:10:16.811711+0000","last_unstale":"2023-10-30T12:32:22.679525+0000","last_undegraded":"2023-10-30T12:32:22.679525+0000","last_fullsized":"2023-10-30T12:32:22.679525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:57.059004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":848,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.687380+0000","last_change":"2023-10-30T12:12:02.414741+0000","last_active":"2023-10-30T12:32:22.687380+0000","last_peered":"2023-10-30T12:32:22.687380+0000","last_clean":"2023-10-30T12:32:22.687380+0000","last_became_active":"2023-10-30T12:12:02.408646+0000","last_became_peered":"2023-10-30T12:12:02.408646+0000","last_unstale":"2023-10-30T12:32:22.687380+0000","last_undegraded":"2023-10-30T12:32:22.687380+0000","last_fullsized":"2023-10-30T12:32:22.687380+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:48:04.995634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":870,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.723681+0000","last_change":"2023-10-30T12:10:16.879470+0000","last_active":"2023-10-30T12:32:22.723681+0000","last_peered":"2023-10-30T12:32:22.723681+0000","last_clean":"2023-10-30T12:32:22.723681+0000","last_became_active":"2023-10-30T12:10:16.849940+0000","last_became_peered":"2023-10-30T12:10:16.849940+0000","last_unstale":"2023-10-30T12:32:22.723681+0000","last_undegraded":"2023-10-30T12:32:22.723681+0000","last_fullsized":"2023-10-30T12:32:22.723681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:09:07.690364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":857,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.787493+0000","last_change":"2023-10-30T12:11:53.127463+0000","last_active":"2023-10-30T12:32:22.787493+0000","last_peered":"2023-10-30T12:32:22.787493+0000","last_clean":"2023-10-30T12:32:22.787493+0000","last_became_active":"2023-10-30T12:11:53.066917+0000","last_became_peered":"2023-10-30T12:11:53.066917+0000","last_unstale":"2023-10-30T12:32:22.787493+0000","last_undegraded":"2023-10-30T12:32:22.787493+0000","last_fullsized":"2023-10-30T12:32:22.787493+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:38.309311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":870,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.798542+0000","last_change":"2023-10-30T12:10:17.161654+0000","last_active":"2023-10-30T12:32:22.798542+0000","last_peered":"2023-10-30T12:32:22.798542+0000","last_clean":"2023-10-30T12:32:22.798542+0000","last_became_active":"2023-10-30T12:10:17.153120+0000","last_became_peered":"2023-10-30T12:10:17.153120+0000","last_unstale":"2023-10-30T12:32:22.798542+0000","last_undegraded":"2023-10-30T12:32:22.798542+0000","last_fullsized":"2023-10-30T12:32:22.798542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:42.496061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":963,"reported_epoch":894,"state":"active+clean","last_fresh":"2023-10-30T12:32:23.373974+0000","last_change":"2023-10-30T12:10:09.781745+0000","last_active":"2023-10-30T12:32:23.373974+0000","last_peered":"2023-10-30T12:32:23.373974+0000","last_clean":"2023-10-30T12:32:23.373974+0000","last_became_active":"2023-10-30T12:10:09.748373+0000","last_became_peered":"2023-10-30T12:10:09.748373+0000","last_unstale":"2023-10-30T12:32:23.373974+0000","last_undegraded":"2023-10-30T12:32:23.373974+0000","last_fullsized":"2023-10-30T12:32:23.373974+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:07.000545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:07.000545+0000","last_clean_scrub_stamp":"2023-10-30T12:10:07.000545+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:30.943478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":857,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.833655+0000","last_change":"2023-10-30T12:12:03.636874+0000","last_active":"2023-10-30T12:32:22.833655+0000","last_peered":"2023-10-30T12:32:22.833655+0000","last_clean":"2023-10-30T12:32:22.833655+0000","last_became_active":"2023-10-30T12:12:03.631410+0000","last_became_peered":"2023-10-30T12:12:03.631410+0000","last_unstale":"2023-10-30T12:32:22.833655+0000","last_undegraded":"2023-10-30T12:32:22.833655+0000","last_fullsized":"2023-10-30T12:32:22.833655+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:40.711737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":868,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.211102+0000","last_change":"2023-10-30T12:12:01.935080+0000","last_active":"2023-10-30T12:32:26.211102+0000","last_peered":"2023-10-30T12:32:26.211102+0000","last_clean":"2023-10-30T12:32:26.211102+0000","last_became_active":"2023-10-30T12:12:01.926913+0000","last_became_peered":"2023-10-30T12:12:01.926913+0000","last_unstale":"2023-10-30T12:32:26.211102+0000","last_undegraded":"2023-10-30T12:32:26.211102+0000","last_fullsized":"2023-10-30T12:32:26.211102+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:08:28.436909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":848,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.677837+0000","last_change":"2023-10-30T12:12:02.114160+0000","last_active":"2023-10-30T12:32:22.677837+0000","last_peered":"2023-10-30T12:32:22.677837+0000","last_clean":"2023-10-30T12:32:22.677837+0000","last_became_active":"2023-10-30T12:12:02.108733+0000","last_became_peered":"2023-10-30T12:12:02.108733+0000","last_unstale":"2023-10-30T12:32:22.677837+0000","last_undegraded":"2023-10-30T12:32:22.677837+0000","last_fullsized":"2023-10-30T12:32:22.677837+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:57.228464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":856,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:32:24.547606+0000","last_change":"2023-10-30T12:11:52.314662+0000","last_active":"2023-10-30T12:32:24.547606+0000","last_peered":"2023-10-30T12:32:24.547606+0000","last_clean":"2023-10-30T12:32:24.547606+0000","last_became_active":"2023-10-30T12:11:52.304910+0000","last_became_peered":"2023-10-30T12:11:52.304910+0000","last_unstale":"2023-10-30T12:32:24.547606+0000","last_undegraded":"2023-10-30T12:32:24.547606+0000","last_fullsized":"2023-10-30T12:32:24.547606+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:25:49.066874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":862,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.871776+0000","last_change":"2023-10-30T12:12:10.797605+0000","last_active":"2023-10-30T12:32:22.871776+0000","last_peered":"2023-10-30T12:32:22.871776+0000","last_clean":"2023-10-30T12:32:22.871776+0000","last_became_active":"2023-10-30T12:12:10.791950+0000","last_became_peered":"2023-10-30T12:12:10.791950+0000","last_unstale":"2023-10-30T12:32:22.871776+0000","last_undegraded":"2023-10-30T12:32:22.871776+0000","last_fullsized":"2023-10-30T12:32:22.871776+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:18.345944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":868,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.229662+0000","last_change":"2023-10-30T12:11:56.058505+0000","last_active":"2023-10-30T12:32:26.229662+0000","last_peered":"2023-10-30T12:32:26.229662+0000","last_clean":"2023-10-30T12:32:26.229662+0000","last_became_active":"2023-10-30T12:11:56.049790+0000","last_became_peered":"2023-10-30T12:11:56.049790+0000","last_unstale":"2023-10-30T12:32:26.229662+0000","last_undegraded":"2023-10-30T12:32:26.229662+0000","last_fullsized":"2023-10-30T12:32:26.229662+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:10.872206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":848,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.714566+0000","last_change":"2023-10-30T12:12:02.443381+0000","last_active":"2023-10-30T12:32:22.714566+0000","last_peered":"2023-10-30T12:32:22.714566+0000","last_clean":"2023-10-30T12:32:22.714566+0000","last_became_active":"2023-10-30T12:12:02.435431+0000","last_became_peered":"2023-10-30T12:12:02.435431+0000","last_unstale":"2023-10-30T12:32:22.714566+0000","last_undegraded":"2023-10-30T12:32:22.714566+0000","last_fullsized":"2023-10-30T12:32:22.714566+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:04.786017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":848,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.698057+0000","last_change":"2023-10-30T12:11:51.298662+0000","last_active":"2023-10-30T12:32:22.698057+0000","last_peered":"2023-10-30T12:32:22.698057+0000","last_clean":"2023-10-30T12:32:22.698057+0000","last_became_active":"2023-10-30T12:11:51.289446+0000","last_became_peered":"2023-10-30T12:11:51.289446+0000","last_unstale":"2023-10-30T12:32:22.698057+0000","last_undegraded":"2023-10-30T12:32:22.698057+0000","last_fullsized":"2023-10-30T12:32:22.698057+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:03:40.836674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":849,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.703055+0000","last_change":"2023-10-30T12:11:53.122439+0000","last_active":"2023-10-30T12:32:22.703055+0000","last_peered":"2023-10-30T12:32:22.703055+0000","last_clean":"2023-10-30T12:32:22.703055+0000","last_became_active":"2023-10-30T12:11:53.110390+0000","last_became_peered":"2023-10-30T12:11:53.110390+0000","last_unstale":"2023-10-30T12:32:22.703055+0000","last_undegraded":"2023-10-30T12:32:22.703055+0000","last_fullsized":"2023-10-30T12:32:22.703055+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:52.930583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":867,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.133886+0000","last_change":"2023-10-30T12:12:00.804118+0000","last_active":"2023-10-30T12:32:26.133886+0000","last_peered":"2023-10-30T12:32:26.133886+0000","last_clean":"2023-10-30T12:32:26.133886+0000","last_became_active":"2023-10-30T12:12:00.784060+0000","last_became_peered":"2023-10-30T12:12:00.784060+0000","last_unstale":"2023-10-30T12:32:26.133886+0000","last_undegraded":"2023-10-30T12:32:26.133886+0000","last_fullsized":"2023-10-30T12:32:26.133886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:58:44.332584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":867,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.131879+0000","last_change":"2023-10-30T12:12:01.684690+0000","last_active":"2023-10-30T12:32:26.131879+0000","last_peered":"2023-10-30T12:32:26.131879+0000","last_clean":"2023-10-30T12:32:26.131879+0000","last_became_active":"2023-10-30T12:12:01.679359+0000","last_became_peered":"2023-10-30T12:12:01.679359+0000","last_unstale":"2023-10-30T12:32:26.131879+0000","last_undegraded":"2023-10-30T12:32:26.131879+0000","last_fullsized":"2023-10-30T12:32:26.131879+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:14.805349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":863,"reported_epoch":893,"state":"active+clean","last_fresh":"2023-10-30T12:32:22.722020+0000","last_change":"2023-10-30T12:12:08.288834+0000","last_active":"2023-10-30T12:32:22.722020+0000","last_peered":"2023-10-30T12:32:22.722020+0000","last_clean":"2023-10-30T12:32:22.722020+0000","last_became_active":"2023-10-30T12:12:08.259922+0000","last_became_peered":"2023-10-30T12:12:08.259922+0000","last_unstale":"2023-10-30T12:32:22.722020+0000","last_undegraded":"2023-10-30T12:32:22.722020+0000","last_fullsized":"2023-10-30T12:32:22.722020+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:47:09.939780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477006,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78420,"kb_used_data":6972,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293420,"statfs":{"total":96636764160,"available":96556462080,"internally_reserved":0,"allocated":7139328,"data_stored":3429418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73136584},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":68,"apply_latency_ms":68,"commit_latency_ns":68000000,"apply_latency_ns":68000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:55 2023","interfaces":[{"interface":"back","average":{"1min":10.798999999999999,"5min":17.038,"15min":20.966000000000001},"min":{"1min":6.4589999999999996,"5min":4.1319999999999997,"15min":4.1319999999999997},"max":{"1min":26.298999999999999,"5min":140.75200000000001,"15min":213.76499999999999},"last":6.4779999999999998},{"interface":"front","average":{"1min":13.866,"5min":19.09,"15min":24.885999999999999},"min":{"1min":5.6989999999999998,"5min":4.21,"15min":4.1360000000000001},"max":{"1min":26.111999999999998,"5min":142.81200000000001,"15min":265.09100000000001},"last":12.125}]},{"osd":1,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":16.192,"5min":19.501000000000001,"15min":21.971},"min":{"1min":6.9690000000000003,"5min":5.3449999999999998,"15min":4.6609999999999996},"max":{"1min":60.024999999999999,"5min":119.535,"15min":164.755},"last":8.3870000000000005},{"interface":"front","average":{"1min":16.702999999999999,"5min":19.878,"15min":25.541},"min":{"1min":7.5990000000000002,"5min":6.6399999999999997,"15min":6.0990000000000002},"max":{"1min":72.597999999999999,"5min":146.874,"15min":260.19099999999997},"last":10.602}]},{"osd":2,"last update":"Mon Oct 30 12:31:55 2023","interfaces":[{"interface":"back","average":{"1min":18.702000000000002,"5min":23.050999999999998,"15min":32.445},"min":{"1min":9.2129999999999992,"5min":6.4589999999999996,"15min":6.4589999999999996},"max":{"1min":98.338999999999999,"5min":215.48400000000001,"15min":277.137},"last":13.444000000000001},{"interface":"front","average":{"1min":17.099,"5min":23.308,"15min":32.445},"min":{"1min":9.9890000000000008,"5min":6.5860000000000003,"15min":6.5860000000000003},"max":{"1min":74.525999999999996,"5min":254.81999999999999,"15min":265.447},"last":12.789}]},{"osd":4,"last update":"Mon Oct 30 12:31:55 2023","interfaces":[{"interface":"back","average":{"1min":14.472,"5min":22.044,"15min":30.475999999999999},"min":{"1min":8.7599999999999998,"5min":8.7599999999999998,"15min":7.8780000000000001},"max":{"1min":23.257999999999999,"5min":139.517,"15min":287.06200000000001},"last":16.387},{"interface":"front","average":{"1min":17.981999999999999,"5min":24.353000000000002,"15min":34.198999999999998},"min":{"1min":11.537000000000001,"5min":11.537000000000001,"15min":10.587},"max":{"1min":28.649000000000001,"5min":130.221,"15min":259.279},"last":19.170999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":20.829000000000001,"5min":28.876999999999999,"15min":35.593000000000004},"min":{"1min":12.195,"5min":11.795,"15min":8.2550000000000008},"max":{"1min":74.197999999999993,"5min":324.26600000000002,"15min":324.26600000000002},"last":14.180999999999999},{"interface":"front","average":{"1min":15.435,"5min":19.82,"15min":24.088000000000001},"min":{"1min":7.1639999999999997,"5min":6.6900000000000004,"15min":6.5679999999999996},"max":{"1min":70.387,"5min":251.57900000000001,"15min":251.57900000000001},"last":9.141}]},{"osd":6,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":21.481999999999999,"5min":28.734000000000002,"15min":28.196999999999999},"min":{"1min":12.811999999999999,"5min":10.584,"15min":7.9370000000000003},"max":{"1min":68.438999999999993,"5min":135.005,"15min":218.80600000000001},"last":14.81},{"interface":"front","average":{"1min":13.765000000000001,"5min":20.492000000000001,"15min":27.355},"min":{"1min":9.7479999999999993,"5min":8.0120000000000005,"15min":6.8170000000000002},"max":{"1min":25.957999999999998,"5min":133.999,"15min":239.80500000000001},"last":9.8179999999999996}]},{"osd":7,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":22.760999999999999,"5min":33.521999999999998,"15min":40.765000000000001},"min":{"1min":13.420999999999999,"5min":13.420999999999999,"15min":10.132999999999999},"max":{"1min":67.742999999999995,"5min":323.58199999999999,"15min":1077.606},"last":18.309999999999999},{"interface":"front","average":{"1min":15.550000000000001,"5min":26.276,"15min":29.468},"min":{"1min":9.5069999999999997,"5min":9.0050000000000008,"15min":8.5879999999999992},"max":{"1min":21.239000000000001,"5min":253.04599999999999,"15min":253.04599999999999},"last":16.992000000000001}]}]},{"osd":2,"up_from":16,"seq":68719477006,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60344,"kb_used_data":6948,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311496,"statfs":{"total":96636764160,"available":96574971904,"internally_reserved":0,"allocated":7114752,"data_stored":3385335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":76,"apply_latency_ms":76,"commit_latency_ns":76000000,"apply_latency_ns":76000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":22.933,"5min":16.960999999999999,"15min":24.920000000000002},"min":{"1min":7.641,"5min":4.5149999999999997,"15min":4.5149999999999997},"max":{"1min":205.785,"5min":205.785,"15min":398.01499999999999},"last":10.486000000000001},{"interface":"front","average":{"1min":19.140000000000001,"5min":18.173999999999999,"15min":24.411000000000001},"min":{"1min":6.8419999999999996,"5min":5.9969999999999999,"15min":4.4790000000000001},"max":{"1min":135.32499999999999,"5min":135.32499999999999,"15min":376.49299999999999},"last":8.0210000000000008}]},{"osd":1,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":23.818000000000001,"5min":19.024000000000001,"15min":26.135000000000002},"min":{"1min":7.2519999999999998,"5min":7.2519999999999998,"15min":6.9660000000000002},"max":{"1min":172.09999999999999,"5min":172.09999999999999,"15min":317.92099999999999},"last":12.393000000000001},{"interface":"front","average":{"1min":28.353999999999999,"5min":23.835000000000001,"15min":32.107999999999997},"min":{"1min":10.314,"5min":9.7509999999999994,"15min":7.9989999999999997},"max":{"1min":183.803,"5min":183.803,"15min":355.99400000000003},"last":13.548999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":29.100999999999999,"5min":23.119,"15min":29.891999999999999},"min":{"1min":11.067,"5min":9.3870000000000005,"15min":6.4080000000000004},"max":{"1min":207.35300000000001,"5min":207.35300000000001,"15min":412.42899999999997},"last":16.09},{"interface":"front","average":{"1min":20.431000000000001,"5min":17.634,"15min":28.106999999999999},"min":{"1min":6.1959999999999997,"5min":6.1959999999999997,"15min":6.1959999999999997},"max":{"1min":147.63900000000001,"5min":147.63900000000001,"15min":328.90899999999999},"last":7.3929999999999998}]},{"osd":4,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":17.556999999999999,"5min":16.536999999999999,"15min":22.847000000000001},"min":{"1min":8.3580000000000005,"5min":7.556,"15min":6.5439999999999996},"max":{"1min":44.722000000000001,"5min":69.641000000000005,"15min":326.565},"last":200.41300000000001},{"interface":"front","average":{"1min":20.829000000000001,"5min":20.02,"15min":27.216999999999999},"min":{"1min":12.151,"5min":10.289,"15min":9.8710000000000004},"max":{"1min":49.927,"5min":83.744,"15min":302.76499999999999},"last":45.310000000000002}]},{"osd":5,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":23.760999999999999,"5min":19.529,"15min":22.626000000000001},"min":{"1min":11.141,"5min":5.4139999999999997,"15min":4.9539999999999997},"max":{"1min":91.162000000000006,"5min":112.432,"15min":246.011},"last":211.773},{"interface":"front","average":{"1min":13.92,"5min":17.667999999999999,"15min":30.521999999999998},"min":{"1min":7.008,"5min":6.492,"15min":6.492},"max":{"1min":89.206000000000003,"5min":155.58699999999999,"15min":350.80399999999997},"last":208.166}]},{"osd":6,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":20.018000000000001,"5min":21.427,"15min":29.437999999999999},"min":{"1min":10.337999999999999,"5min":9.5139999999999993,"15min":7.8949999999999996},"max":{"1min":47.087000000000003,"5min":94.465000000000003,"15min":374.16699999999997},"last":170.02799999999999},{"interface":"front","average":{"1min":26.401,"5min":24.434000000000001,"15min":31.552},"min":{"1min":15.363,"5min":10.032,"15min":8.2400000000000002},"max":{"1min":82.998000000000005,"5min":98.968999999999994,"15min":333.37299999999999},"last":35.454000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":21.491,"5min":15.154,"15min":23.888000000000002},"min":{"1min":8.4600000000000009,"5min":6.8049999999999997,"15min":6.1539999999999999},"max":{"1min":88.486999999999995,"5min":88.486999999999995,"15min":298.68900000000002},"last":9.7690000000000001},{"interface":"front","average":{"1min":26.132000000000001,"5min":19.689,"15min":32.052999999999997},"min":{"1min":9.7449999999999992,"5min":9.7449999999999992,"15min":7.4480000000000004},"max":{"1min":114.895,"5min":114.895,"15min":453.28399999999999},"last":11.118}]}]},{"osd":0,"up_from":16,"seq":68719477005,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60368,"kb_used_data":6976,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311472,"statfs":{"total":96636764160,"available":96574947328,"internally_reserved":0,"allocated":7143424,"data_stored":3494954,"data_compressed":299,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":12.099,"5min":18.317,"15min":20.573},"min":{"1min":6.7759999999999998,"5min":5.6520000000000001,"15min":4.79},"max":{"1min":29.474,"5min":152.46799999999999,"15min":212.69200000000001},"last":7.5979999999999999},{"interface":"front","average":{"1min":11.606999999999999,"5min":17.617000000000001,"15min":20.478000000000002},"min":{"1min":5.1799999999999997,"5min":5.1799999999999997,"15min":5.1799999999999997},"max":{"1min":27.994,"5min":150.90100000000001,"15min":219.46600000000001},"last":8.4109999999999996}]},{"osd":2,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":16.555,"5min":17.678000000000001,"15min":21.452999999999999},"min":{"1min":7.8090000000000002,"5min":5.9580000000000002,"15min":5.9580000000000002},"max":{"1min":80.406999999999996,"5min":120.355,"15min":294.21699999999998},"last":12.138},{"interface":"front","average":{"1min":17.379999999999999,"5min":17.045000000000002,"15min":19.501000000000001},"min":{"1min":8.4209999999999994,"5min":6.5499999999999998,"15min":5.8010000000000002},"max":{"1min":96.001999999999995,"5min":101.89100000000001,"15min":237.929},"last":13.430999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":12.368,"5min":20.166,"15min":19.789000000000001},"min":{"1min":5.484,"5min":5.484,"15min":5.484},"max":{"1min":68.653000000000006,"5min":210.88900000000001,"15min":210.88900000000001},"last":6.1289999999999996},{"interface":"front","average":{"1min":17.181000000000001,"5min":17.488,"15min":21.372},"min":{"1min":7.4690000000000003,"5min":6.2770000000000001,"15min":5.8129999999999997},"max":{"1min":72.388999999999996,"5min":220.589,"15min":220.589},"last":9.048}]},{"osd":4,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":20.411999999999999,"5min":23.530999999999999,"15min":21.559000000000001},"min":{"1min":10.009,"5min":8.8040000000000003,"15min":6.4329999999999998},"max":{"1min":110.872,"5min":247.57300000000001,"15min":247.57300000000001},"last":12.781000000000001},{"interface":"front","average":{"1min":13.563000000000001,"5min":19.027999999999999,"15min":21.359000000000002},"min":{"1min":7.7889999999999997,"5min":6.7370000000000001,"15min":6.7370000000000001},"max":{"1min":23.960999999999999,"5min":267.23899999999998,"15min":267.23899999999998},"last":9.7590000000000003}]},{"osd":5,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":13.832000000000001,"5min":21.916,"15min":21.706},"min":{"1min":7.3899999999999997,"5min":7.3899999999999997,"15min":6.7080000000000002},"max":{"1min":25.125,"5min":232.041,"15min":232.041},"last":15.644},{"interface":"front","average":{"1min":16.873000000000001,"5min":25.713000000000001,"15min":25.507000000000001},"min":{"1min":11.106,"5min":9.843,"15min":9.843},"max":{"1min":27.648,"5min":256.58699999999999,"15min":256.58699999999999},"last":16.501000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":21.965,"5min":23.777999999999999,"15min":24.338999999999999},"min":{"1min":11.445,"5min":8.2750000000000004,"15min":8.2750000000000004},"max":{"1min":116.67100000000001,"5min":136.44399999999999,"15min":235.15199999999999},"last":14.853999999999999},{"interface":"front","average":{"1min":18.707000000000001,"5min":24.120999999999999,"15min":24.725000000000001},"min":{"1min":9.9090000000000007,"5min":8.4619999999999997,"15min":8.4619999999999997},"max":{"1min":56.292000000000002,"5min":196.33799999999999,"15min":196.33799999999999},"last":18.506}]},{"osd":7,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":14.186,"5min":17.329000000000001,"15min":20.686},"min":{"1min":7.851,"5min":7.2110000000000003,"15min":7.2110000000000003},"max":{"1min":20.734000000000002,"5min":95.358999999999995,"15min":232.815},"last":19.132999999999999},{"interface":"front","average":{"1min":19.161000000000001,"5min":22.934000000000001,"15min":26.524000000000001},"min":{"1min":10.509,"5min":9.1110000000000007,"15min":8.827},"max":{"1min":28.800999999999998,"5min":104.599,"15min":199.85300000000001},"last":14.154999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542414,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60360,"kb_used_data":6960,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311480,"statfs":{"total":96636764160,"available":96574955520,"internally_reserved":0,"allocated":7127040,"data_stored":3478570,"data_compressed":299,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":9.0199999999999996,"5min":13.888999999999999,"15min":21.597000000000001},"min":{"1min":4.7370000000000001,"5min":4.1820000000000004,"15min":3.7999999999999998},"max":{"1min":16.635999999999999,"5min":175.166,"15min":410.46699999999998},"last":8.3789999999999996},{"interface":"front","average":{"1min":9.7629999999999999,"5min":14.135999999999999,"15min":23.727},"min":{"1min":4.335,"5min":4.0570000000000004,"15min":4.0570000000000004},"max":{"1min":18.619,"5min":182.875,"15min":1113.894},"last":7.3780000000000001}]},{"osd":1,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":12.548,"5min":14.526,"15min":22.489000000000001},"min":{"1min":5.8879999999999999,"5min":5.1840000000000002,"15min":4.5839999999999996},"max":{"1min":38.268999999999998,"5min":71.530000000000001,"15min":484.42899999999997},"last":9.0129999999999999},{"interface":"front","average":{"1min":11.441000000000001,"5min":16.390000000000001,"15min":24.693999999999999},"min":{"1min":7.0030000000000001,"5min":5.1369999999999996,"15min":4.7300000000000004},"max":{"1min":25.640999999999998,"5min":105.95399999999999,"15min":344.97699999999998},"last":12.894}]},{"osd":2,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":13.994,"5min":19.303999999999998,"15min":26.847000000000001},"min":{"1min":7.8949999999999996,"5min":7.2370000000000001,"15min":5.242},"max":{"1min":23.893000000000001,"5min":179.96600000000001,"15min":443.49000000000001},"last":56.945999999999998},{"interface":"front","average":{"1min":15.140000000000001,"5min":20.611000000000001,"15min":28.088000000000001},"min":{"1min":7.6820000000000004,"5min":5.883,"15min":5.883},"max":{"1min":30.905000000000001,"5min":186.96899999999999,"15min":318.04000000000002},"last":56.183999999999997}]},{"osd":3,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":12.912000000000001,"5min":22.318000000000001,"15min":28.292000000000002},"min":{"1min":7.7729999999999997,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":19.795000000000002,"5min":237.43899999999999,"15min":419.54599999999999},"last":45.630000000000003},{"interface":"front","average":{"1min":16.077999999999999,"5min":24.021000000000001,"15min":30.792999999999999},"min":{"1min":8.3559999999999999,"5min":8.0380000000000003,"15min":7.4969999999999999},"max":{"1min":43.798000000000002,"5min":280.57400000000001,"15min":353.90800000000002},"last":44.710999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":20.606000000000002,"5min":19.699000000000002,"15min":25.931000000000001},"min":{"1min":8.0239999999999991,"5min":6.1529999999999996,"15min":5.7039999999999997},"max":{"1min":125.714,"5min":185.97200000000001,"15min":420.41399999999999},"last":10.194000000000001},{"interface":"front","average":{"1min":21.055,"5min":21.521000000000001,"15min":27.934000000000001},"min":{"1min":7.1020000000000003,"5min":6.6799999999999997,"15min":6.6799999999999997},"max":{"1min":120.68000000000001,"5min":214.81,"15min":329.49000000000001},"last":12.285}]},{"osd":5,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":14.662000000000001,"5min":19.605,"15min":28.992000000000001},"min":{"1min":8.8040000000000003,"5min":8.4689999999999994,"15min":7.6449999999999996},"max":{"1min":27.105,"5min":172.79300000000001,"15min":449.233},"last":10.99},{"interface":"front","average":{"1min":20.103000000000002,"5min":24.542999999999999,"15min":33.482999999999997},"min":{"1min":11.194000000000001,"5min":11.194000000000001,"15min":10.382},"max":{"1min":47.503999999999998,"5min":173.39699999999999,"15min":347.05099999999999},"last":13.617000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":15.154999999999999,"5min":21.318000000000001,"15min":27.535},"min":{"1min":8.4199999999999999,"5min":8.2230000000000008,"15min":6.6059999999999999},"max":{"1min":19.495000000000001,"5min":240.779,"15min":454.25400000000002},"last":11.680999999999999},{"interface":"front","average":{"1min":19.113,"5min":24.262,"15min":29.777000000000001},"min":{"1min":12.965999999999999,"5min":9.8710000000000004,"15min":9.8710000000000004},"max":{"1min":42.076000000000001,"5min":198.68199999999999,"15min":507.48700000000002},"last":15.108000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542413,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100596,"kb_used_data":7004,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94271244,"statfs":{"total":96636764160,"available":96533753856,"internally_reserved":0,"allocated":7172096,"data_stored":3813577,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":95812038},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":14.826000000000001,"5min":15.159000000000001,"15min":21.579999999999998},"min":{"1min":5.1609999999999996,"5min":5.1609999999999996,"15min":4.1980000000000004},"max":{"1min":64.741,"5min":92.221999999999994,"15min":431.12},"last":8.9489999999999998},{"interface":"front","average":{"1min":17.555,"5min":15.711,"15min":21.766999999999999},"min":{"1min":6.2409999999999997,"5min":5.1420000000000003,"15min":4.7199999999999998},"max":{"1min":98.355999999999995,"5min":98.355999999999995,"15min":399.90300000000002},"last":8.032}]},{"osd":2,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":26.707000000000001,"5min":23.045999999999999,"15min":27.221},"min":{"1min":6.7359999999999998,"5min":6.7359999999999998,"15min":6.3339999999999996},"max":{"1min":162.07900000000001,"5min":162.07900000000001,"15min":284.32900000000001},"last":40.326000000000001},{"interface":"front","average":{"1min":21.553000000000001,"5min":19.859000000000002,"15min":25.832000000000001},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":5.8230000000000004},"max":{"1min":118.956,"5min":118.956,"15min":331.928},"last":21.725000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":23.033999999999999,"5min":24.411000000000001,"15min":31.416},"min":{"1min":8.1790000000000003,"5min":7.1970000000000001,"15min":6.0229999999999997},"max":{"1min":104.30500000000001,"5min":153.827,"15min":270.78100000000001},"last":23.405999999999999},{"interface":"front","average":{"1min":21.942,"5min":24.722000000000001,"15min":29.132000000000001},"min":{"1min":8.5429999999999993,"5min":6.633,"15min":6.4960000000000004},"max":{"1min":106.015,"5min":235.411,"15min":321.42700000000002},"last":19.02}]},{"osd":4,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":22.050000000000001,"5min":26.515000000000001,"15min":27.753},"min":{"1min":10.422000000000001,"5min":8.4659999999999993,"15min":6.6890000000000001},"max":{"1min":143.80600000000001,"5min":196.18000000000001,"15min":278.36399999999998},"last":18.353999999999999},{"interface":"front","average":{"1min":21.295999999999999,"5min":24.879999999999999,"15min":25.135999999999999},"min":{"1min":8.8339999999999996,"5min":7.641,"15min":6.3280000000000003},"max":{"1min":148.40100000000001,"5min":219.45400000000001,"15min":284.76900000000001},"last":9.7390000000000008}]},{"osd":5,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":21.213999999999999,"5min":21.710999999999999,"15min":26.812999999999999},"min":{"1min":7.077,"5min":6.3140000000000001,"15min":6.0869999999999997},"max":{"1min":117.886,"5min":127.604,"15min":346.97300000000001},"last":10.347},{"interface":"front","average":{"1min":22.224,"5min":26.302,"15min":29.582000000000001},"min":{"1min":10.473000000000001,"5min":9.8249999999999993,"15min":8.2370000000000001},"max":{"1min":110.979,"5min":280.072,"15min":313.72899999999998},"last":15.576000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":21.413,"5min":24.916,"15min":30.504999999999999},"min":{"1min":8.9939999999999998,"5min":6.7089999999999996,"15min":6.7089999999999996},"max":{"1min":126.279,"5min":126.279,"15min":391.726},"last":11.976000000000001},{"interface":"front","average":{"1min":25.158999999999999,"5min":24.603000000000002,"15min":27.167000000000002},"min":{"1min":8.7750000000000004,"5min":8.2289999999999992,"15min":8.0549999999999997},"max":{"1min":149.857,"5min":187.96799999999999,"15min":288.44099999999997},"last":11.074999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":25.628,"5min":25.503,"15min":39.037999999999997},"min":{"1min":12.448,"5min":8.6899999999999995,"15min":8.6899999999999995},"max":{"1min":137.25299999999999,"5min":137.25299999999999,"15min":1115.931},"last":14.255000000000001},{"interface":"front","average":{"1min":29.222999999999999,"5min":32.040999999999997,"15min":39.862000000000002},"min":{"1min":13.659000000000001,"5min":10.536,"15min":9.5079999999999991},"max":{"1min":151.57300000000001,"5min":199.08099999999999,"15min":1115.2660000000001},"last":16.552}]}]},{"osd":6,"up_from":13,"seq":55834575117,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60372,"kb_used_data":6972,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311468,"statfs":{"total":96636764160,"available":96574943232,"internally_reserved":0,"allocated":7139328,"data_stored":3397051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":54,"apply_latency_ms":54,"commit_latency_ns":54000000,"apply_latency_ns":54000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:58 2023","interfaces":[{"interface":"back","average":{"1min":9.2989999999999995,"5min":14.662000000000001,"15min":15.426},"min":{"1min":4.0119999999999996,"5min":3.9380000000000002,"15min":3.9380000000000002},"max":{"1min":36.322000000000003,"5min":200.376,"15min":220.25700000000001},"last":110.438},{"interface":"front","average":{"1min":9.7750000000000004,"5min":16.427,"15min":15.846},"min":{"1min":5.4180000000000001,"5min":4.6449999999999996,"15min":4.1799999999999997},"max":{"1min":28.044,"5min":196.261,"15min":219.40000000000001},"last":90.683999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:31:51 2023","interfaces":[{"interface":"back","average":{"1min":13.768000000000001,"5min":18.015000000000001,"15min":18.302},"min":{"1min":7.6390000000000002,"5min":5.1180000000000003,"15min":4.306},"max":{"1min":44.209000000000003,"5min":215.86699999999999,"15min":216.90299999999999},"last":131.17500000000001},{"interface":"front","average":{"1min":15.76,"5min":19.614999999999998,"15min":18.779},"min":{"1min":7.0019999999999998,"5min":5.2039999999999997,"15min":4.5570000000000004},"max":{"1min":42.402000000000001,"5min":225.39699999999999,"15min":225.39699999999999},"last":143.58799999999999}]},{"osd":2,"last update":"Mon Oct 30 12:32:07 2023","interfaces":[{"interface":"back","average":{"1min":12.401,"5min":18.358000000000001,"15min":19.98},"min":{"1min":6.3170000000000002,"5min":5.4729999999999999,"15min":5.4729999999999999},"max":{"1min":93.713999999999999,"5min":245.411,"15min":245.411},"last":11.587999999999999},{"interface":"front","average":{"1min":14.782999999999999,"5min":22.184000000000001,"15min":22.800000000000001},"min":{"1min":8.9339999999999993,"5min":6.7679999999999998,"15min":6.7679999999999998},"max":{"1min":26.739999999999998,"5min":244.76400000000001,"15min":244.76400000000001},"last":15.132999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:31:58 2023","interfaces":[{"interface":"back","average":{"1min":14.455,"5min":25.152000000000001,"15min":22.244},"min":{"1min":7.1970000000000001,"5min":5.9649999999999999,"15min":5.9649999999999999},"max":{"1min":35.399999999999999,"5min":217.495,"15min":300.44600000000003},"last":6.7670000000000003},{"interface":"front","average":{"1min":17.216999999999999,"5min":26.016999999999999,"15min":23.670000000000002},"min":{"1min":10.24,"5min":9.6560000000000006,"15min":8.4960000000000004},"max":{"1min":31.969000000000001,"5min":249.18100000000001,"15min":249.18100000000001},"last":10.657999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:31:51 2023","interfaces":[{"interface":"back","average":{"1min":15.673,"5min":21.34,"15min":23.218},"min":{"1min":9.9610000000000003,"5min":8.75,"15min":6.556},"max":{"1min":34.320999999999998,"5min":246.57599999999999,"15min":246.57599999999999},"last":15.901999999999999},{"interface":"front","average":{"1min":21.498000000000001,"5min":20.198,"15min":19.931000000000001},"min":{"1min":7.4119999999999999,"5min":6.9870000000000001,"15min":6.5209999999999999},"max":{"1min":78.716999999999999,"5min":233.952,"15min":233.952},"last":12.555999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:31:34 2023","interfaces":[{"interface":"back","average":{"1min":16.387,"5min":23.277000000000001,"15min":25.347999999999999},"min":{"1min":9.3940000000000001,"5min":9.3940000000000001,"15min":7.4080000000000004},"max":{"1min":24.803000000000001,"5min":267.37799999999999,"15min":348.08699999999999},"last":10},{"interface":"front","average":{"1min":16.57,"5min":20.597000000000001,"15min":23.638000000000002},"min":{"1min":10.117000000000001,"5min":7.0170000000000003,"15min":5.9089999999999998},"max":{"1min":33.296999999999997,"5min":251.63499999999999,"15min":350.89999999999998},"last":8.3149999999999995}]},{"osd":7,"last update":"Mon Oct 30 12:31:34 2023","interfaces":[{"interface":"back","average":{"1min":20.010000000000002,"5min":26.344999999999999,"15min":25.242999999999999},"min":{"1min":11.276,"5min":7.7480000000000002,"15min":5.7359999999999998},"max":{"1min":42.716999999999999,"5min":240.048,"15min":342.39400000000001},"last":142.589},{"interface":"front","average":{"1min":19.148,"5min":25.388000000000002,"15min":25.414000000000001},"min":{"1min":8.1639999999999997,"5min":7.4089999999999998,"15min":7.4089999999999998},"max":{"1min":56.218000000000004,"5min":250.70400000000001,"15min":343.60300000000001},"last":166.40100000000001}]}]},{"osd":5,"up_from":13,"seq":55834575116,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78420,"kb_used_data":6980,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293420,"statfs":{"total":96636764160,"available":96556462080,"internally_reserved":0,"allocated":7147520,"data_stored":3798586,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":61,"apply_latency_ms":61,"commit_latency_ns":61000000,"apply_latency_ns":61000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":13.532999999999999,"5min":15.859999999999999,"15min":16.879999999999999},"min":{"1min":4.6799999999999997,"5min":4.0190000000000001,"15min":3.9620000000000002},"max":{"1min":54.750999999999998,"5min":128.709,"15min":439.73200000000003},"last":7.0999999999999996},{"interface":"front","average":{"1min":12.602,"5min":15.343999999999999,"15min":15.458},"min":{"1min":6.1959999999999997,"5min":5.1280000000000001,"15min":4.3170000000000002},"max":{"1min":54.161000000000001,"5min":128,"15min":180.74700000000001},"last":18.824000000000002}]},{"osd":1,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":20.91,"5min":21.420000000000002,"15min":20.076000000000001},"min":{"1min":5.9880000000000004,"5min":5.8789999999999996,"15min":5.0410000000000004},"max":{"1min":73.456999999999994,"5min":213.06700000000001,"15min":235.05199999999999},"last":12.401},{"interface":"front","average":{"1min":19.178999999999998,"5min":20.559999999999999,"15min":19.687999999999999},"min":{"1min":7.2389999999999999,"5min":5.726,"15min":5.2210000000000001},"max":{"1min":84.025000000000006,"5min":187.249,"15min":420.82900000000001},"last":20.645}]},{"osd":2,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":16.927,"5min":17.652999999999999,"15min":20.605},"min":{"1min":7.4199999999999999,"5min":6.4809999999999999,"15min":5.9539999999999997},"max":{"1min":61.972000000000001,"5min":76.231999999999999,"15min":219.636},"last":23.039000000000001},{"interface":"front","average":{"1min":11.673,"5min":14.134,"15min":18.402000000000001},"min":{"1min":5.3739999999999997,"5min":4.7450000000000001,"15min":4.7450000000000001},"max":{"1min":55.536000000000001,"5min":104.125,"15min":235.387},"last":15.132999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:31:59 2023","interfaces":[{"interface":"back","average":{"1min":13.568,"5min":17.634,"15min":21.957999999999998},"min":{"1min":6.3159999999999998,"5min":6.2089999999999996,"15min":5.0739999999999998},"max":{"1min":52.534999999999997,"5min":110.496,"15min":234.13999999999999},"last":17.047999999999998},{"interface":"front","average":{"1min":17.343,"5min":19.657,"15min":24.550999999999998},"min":{"1min":8.4339999999999993,"5min":6.8070000000000004,"15min":6.5330000000000004},"max":{"1min":58.048000000000002,"5min":135.05500000000001,"15min":232.31700000000001},"last":14.545}]},{"osd":4,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":17.914000000000001,"5min":19.361999999999998,"15min":21.667999999999999},"min":{"1min":7.6159999999999997,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":59.853999999999999,"5min":200.67599999999999,"15min":265.50400000000002},"last":21.468},{"interface":"front","average":{"1min":17.628,"5min":18.516999999999999,"15min":21.404},"min":{"1min":6.3600000000000003,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":58.649000000000001,"5min":108.554,"15min":280.72000000000003},"last":13.028}]},{"osd":6,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":21.899999999999999,"5min":20.135000000000002,"15min":23.164000000000001},"min":{"1min":11.425000000000001,"5min":8.1229999999999993,"15min":8.1229999999999993},"max":{"1min":76.626999999999995,"5min":86.918000000000006,"15min":421.87599999999998},"last":22.265000000000001},{"interface":"front","average":{"1min":20,"5min":22.122,"15min":23.684000000000001},"min":{"1min":8.8409999999999993,"5min":7.3460000000000001,"15min":7.3460000000000001},"max":{"1min":61.104999999999997,"5min":182.376,"15min":353.63099999999997},"last":13.711}]},{"osd":7,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":21.384,"5min":23.309000000000001,"15min":25.530000000000001},"min":{"1min":12.462999999999999,"5min":9.4359999999999999,"15min":6.96},"max":{"1min":59.232999999999997,"5min":124.621,"15min":854.64300000000003},"last":19.658999999999999},{"interface":"front","average":{"1min":13.034000000000001,"5min":17.204000000000001,"15min":19.34},"min":{"1min":7.5110000000000001,"5min":6.2670000000000003,"15min":5.5300000000000002},"max":{"1min":23.940000000000001,"5min":107.976,"15min":264.84699999999998},"last":17.785}]}]},{"osd":4,"up_from":13,"seq":55834575117,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82560,"kb_used_data":6960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289280,"statfs":{"total":96636764160,"available":96552222720,"internally_reserved":0,"allocated":7127040,"data_stored":3391610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":11.680999999999999,"5min":17.094000000000001,"15min":21.251999999999999},"min":{"1min":6.5650000000000004,"5min":5.1920000000000002,"15min":3.8650000000000002},"max":{"1min":35.334000000000003,"5min":166.20400000000001,"15min":372.52300000000002},"last":118.92400000000001},{"interface":"front","average":{"1min":14.744,"5min":20.887,"15min":24.428999999999998},"min":{"1min":8.5109999999999992,"5min":6.0110000000000001,"15min":4.9660000000000002},"max":{"1min":30.943000000000001,"5min":167.251,"15min":200.37299999999999},"last":144.375}]},{"osd":1,"last update":"Mon Oct 30 12:31:31 2023","interfaces":[{"interface":"back","average":{"1min":16.497,"5min":19.984000000000002,"15min":22.559999999999999},"min":{"1min":6.9900000000000002,"5min":6.9900000000000002,"15min":5.7850000000000001},"max":{"1min":88.116,"5min":200.69300000000001,"15min":231.49799999999999},"last":126.119},{"interface":"front","average":{"1min":12.497999999999999,"5min":16.387,"15min":20.524999999999999},"min":{"1min":5.4790000000000001,"5min":4.641,"15min":4.641},"max":{"1min":86.438999999999993,"5min":223.441,"15min":223.441},"last":112.584}]},{"osd":2,"last update":"Mon Oct 30 12:31:31 2023","interfaces":[{"interface":"back","average":{"1min":20.100999999999999,"5min":26.134,"15min":28.177},"min":{"1min":9.3010000000000002,"5min":6.2149999999999999,"15min":6.2149999999999999},"max":{"1min":90.870000000000005,"5min":198.66999999999999,"15min":438.15499999999997},"last":11.478999999999999},{"interface":"front","average":{"1min":17.449000000000002,"5min":19.224,"15min":22.896000000000001},"min":{"1min":6.2409999999999997,"5min":6.2220000000000004,"15min":4.944},"max":{"1min":83.019999999999996,"5min":254.06399999999999,"15min":254.06399999999999},"last":6.976}]},{"osd":3,"last update":"Mon Oct 30 12:32:04 2023","interfaces":[{"interface":"back","average":{"1min":17.190999999999999,"5min":23.48,"15min":28.890000000000001},"min":{"1min":10.754,"5min":6.9710000000000001,"15min":6.9710000000000001},"max":{"1min":52.381999999999998,"5min":269.06700000000001,"15min":269.06700000000001},"last":12.909000000000001},{"interface":"front","average":{"1min":11.754,"5min":16.59,"15min":22.030999999999999},"min":{"1min":4.8230000000000004,"5min":4.8230000000000004,"15min":4.8230000000000004},"max":{"1min":53.374000000000002,"5min":239.715,"15min":239.715},"last":8.6170000000000009}]},{"osd":5,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":16.629000000000001,"5min":23.850000000000001,"15min":24.184999999999999},"min":{"1min":10.301,"5min":8.0670000000000002,"15min":8.0039999999999996},"max":{"1min":38.497,"5min":268.34199999999998,"15min":268.34199999999998},"last":20.835000000000001},{"interface":"front","average":{"1min":15.297000000000001,"5min":20.027000000000001,"15min":21.085000000000001},"min":{"1min":8.8490000000000002,"5min":6.1379999999999999,"15min":5.452},"max":{"1min":36.853000000000002,"5min":252.899,"15min":252.899},"last":9.4480000000000004}]},{"osd":6,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":17.149000000000001,"5min":20.524999999999999,"15min":24.902000000000001},"min":{"1min":8.9629999999999992,"5min":8.8559999999999999,"15min":7.21},"max":{"1min":51.32,"5min":273.90100000000001,"15min":273.90100000000001},"last":12.343},{"interface":"front","average":{"1min":8.6199999999999992,"5min":20.821000000000002,"15min":28.239000000000001},"min":{"1min":5.8849999999999998,"5min":5.8849999999999998,"15min":5.8849999999999998},"max":{"1min":13.861000000000001,"5min":292.11099999999999,"15min":292.11099999999999},"last":10.472}]},{"osd":7,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":17.405999999999999,"5min":25.300000000000001,"15min":30.454999999999998},"min":{"1min":11.175000000000001,"5min":7.4740000000000002,"15min":6.46},"max":{"1min":38.466000000000001,"5min":239.166,"15min":652.50699999999995},"last":185.114},{"interface":"front","average":{"1min":21.138000000000002,"5min":24.059999999999999,"15min":27.285},"min":{"1min":12.268000000000001,"5min":10.422000000000001,"15min":8.9309999999999992},"max":{"1min":114.399,"5min":303.19999999999999,"15min":303.19999999999999},"last":132.596}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:32:29.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:32:30.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.007+0000 7f3372973640 10 monclient: get_monmap_and_config 2023-10-30T12:32:30.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.007+0000 7f3372973640 10 monclient: build_initial_monmap 2023-10-30T12:32:30.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.007+0000 7f3372973640 10 monclient: monmap: 2023-10-30T12:32:30.009 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:30.009 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:30.009 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:30.008161+0000 2023-10-30T12:32:30.010 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:30.008161+0000 2023-10-30T12:32:30.010 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:30.010 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:30.010 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:30.010 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:30.011 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:30.011 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:30.011 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:30.011 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:30.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.007+0000 7f3372973640 1 Processor -- start 2023-10-30T12:32:30.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.007+0000 7f3372973640 1 -- start start 2023-10-30T12:32:30.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.007+0000 7f3372973640 10 monclient: init 2023-10-30T12:32:30.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:30.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-10-30T12:32:30.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f336c0b4390 0x7f336c0a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:30.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 10 monclient(hunting): picked mon.noname-b con 0x7f336c0b4390 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:30.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f336c1418a0 0x7f336c141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:30.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 10 monclient(hunting): picked mon.noname-c con 0x7f336c1418a0 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:30.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f336c0b1b00 0x7f336c0b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:30.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 10 monclient(hunting): picked mon.noname-a con 0x7f336c0b1b00 addr v2:172.21.15.3:3300/0 2023-10-30T12:32:30.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:30.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f336c067a10 con 0x7f336c0b1b00 2023-10-30T12:32:30.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:30.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f336c0b4770 con 0x7f336c1418a0 2023-10-30T12:32:30.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:30.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f336c0a8cb0 con 0x7f336c0b4390 2023-10-30T12:32:30.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:30.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.008+0000 7f3372973640 10 monclient(hunting): authenticate will time out at 2741.222693s 2023-10-30T12:32:30.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f336bfff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f336c0b4390 0x7f336c0a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:30.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f3370f10640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f336c0b1b00 0x7f336c0b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:30.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f3370f10640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f336c0b1b00 0x7f336c0b1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56618/0 (socket says 172.21.15.3:56618) 2023-10-30T12:32:30.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f3370f10640 1 -- 172.21.15.3:0/1855972561 learned_addr learned my addr 172.21.15.3:0/1855972561 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:30.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f3370f10640 10 monclient(hunting): get_auth_request con 0x7f336c0b1b00 auth_method 0 2023-10-30T12:32:30.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f3370f10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:30.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f3370f10640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:30.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f3370f10640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:30.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f336b7fe640 1 --2- 172.21.15.3:0/1855972561 >> v2:172.21.15.92:3300/0 conn(0x7f336c1418a0 0x7f336c141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:30.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f336bfff640 10 monclient(hunting): get_auth_request con 0x7f336c0b4390 auth_method 0 2023-10-30T12:32:30.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f336bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:30.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f336bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:30.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f336bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:30.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f336b7fe640 10 monclient(hunting): get_auth_request con 0x7f336c1418a0 auth_method 0 2023-10-30T12:32:30.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f336b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:30.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f336b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:30.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.009+0000 7f336b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:30.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.010+0000 7f3370f10640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:30.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.010+0000 7f3370f10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:30.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.010+0000 7f3370f10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:30.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.010+0000 7f336bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:30.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.010+0000 7f336bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:30.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.010+0000 7f336bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:30.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.010+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:30.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.010+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:30.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.010+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:30.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.011+0000 7f3370f10640 10 monclient(hunting): handle_auth_done global_id 16599 payload 274 2023-10-30T12:32:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.011+0000 7f3370f10640 1 -- 172.21.15.3:0/1855972561 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f336c0b4390 msgr2=0x7f336c0a8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.011+0000 7f3370f10640 1 --2- 172.21.15.3:0/1855972561 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f336c0b4390 0x7f336c0a8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.011+0000 7f3370f10640 1 -- 172.21.15.3:0/1855972561 >> v2:172.21.15.92:3300/0 conn(0x7f336c1418a0 msgr2=0x7f336c141ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.011+0000 7f3370f10640 1 --2- 172.21.15.3:0/1855972561 >> v2:172.21.15.92:3300/0 conn(0x7f336c1418a0 0x7f336c141ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.011+0000 7f3370f10640 10 monclient: _finish_hunting 0 2023-10-30T12:32:30.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.011+0000 7f3370f10640 1 monclient: found mon.noname-a 2023-10-30T12:32:30.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.011+0000 7f3370f10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:30.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.011+0000 7f3370f10640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:32:30.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.011+0000 7f3370f10640 1 -- 172.21.15.3:0/1855972561 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c0aa3d0 con 0x7f336c0b1b00 2023-10-30T12:32:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.013+0000 7f3370f10640 1 --2- 172.21.15.3:0/1855972561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f336c0b1b00 0x7f336c0b1f90 secure :-1 s=READY pgs=3600 cs=0 l=1 rev1=1 crypto rx=0x7f336c0b8210 tx=0x7f336402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dbdc12e12b6be3b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f336affd640 1 -- 172.21.15.3:0/1855972561 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336402ca70 con 0x7f336c0b1b00 2023-10-30T12:32:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f336affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f336affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:32:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f336affd640 10 monclient: dump: 2023-10-30T12:32:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:30.030 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.030 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.030 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:30.030 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:30.030 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:30.031 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:30.031 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:30.031 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:30.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f336affd640 10 monclient: _finish_auth 0 2023-10-30T12:32:30.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f336affd640 10 monclient: _check_auth_tickets 2023-10-30T12:32:30.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f336affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:30.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f336affd640 1 -- 172.21.15.3:0/1855972561 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336402cbe0 con 0x7f336c0b1b00 2023-10-30T12:32:30.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f336affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:30.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f336affd640 1 -- 172.21.15.3:0/1855972561 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33640343f0 con 0x7f336c0b1b00 2023-10-30T12:32:30.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f3372973640 5 monclient: authenticate success, global_id 16599 2023-10-30T12:32:30.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f3372973640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:30.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f3372973640 4 set_mon_vals no callback set 2023-10-30T12:32:30.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f336affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:30.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f336affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:32:30.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.017+0000 7f336affd640 10 monclient: dump: 2023-10-30T12:32:30.034 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:30.034 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:30.034 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.034 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.034 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:30.034 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:30.035 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:30.035 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:30.035 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:30.035 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:30.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.018+0000 7f3372973640 10 monclient: shutdown 2023-10-30T12:32:30.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.018+0000 7f3372973640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:30.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.018+0000 7f3372973640 1 -- 172.21.15.3:0/1855972561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f336c0b1b00 msgr2=0x7f336c0b1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:30.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.018+0000 7f3372973640 1 --2- 172.21.15.3:0/1855972561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f336c0b1b00 0x7f336c0b1f90 secure :-1 s=READY pgs=3600 cs=0 l=1 rev1=1 crypto rx=0x7f336c0b8210 tx=0x7f336402bed0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.018+0000 7f3372973640 1 -- 172.21.15.3:0/1855972561 shutdown_connections 2023-10-30T12:32:30.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.018+0000 7f3372973640 1 --2- 172.21.15.3:0/1855972561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f336c0b1b00 0x7f336c0b1f90 unknown :-1 s=CLOSED pgs=3600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.018+0000 7f3372973640 1 --2- 172.21.15.3:0/1855972561 >> v2:172.21.15.92:3300/0 conn(0x7f336c1418a0 0x7f336c141ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.018+0000 7f3372973640 1 --2- 172.21.15.3:0/1855972561 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f336c0b4390 0x7f336c0a8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.018+0000 7f3372973640 1 -- 172.21.15.3:0/1855972561 >> 172.21.15.3:0/1855972561 conn(0x7f336c065ea0 msgr2=0x7f336c0662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:30.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.018+0000 7f3372973640 1 -- 172.21.15.3:0/1855972561 shutdown_connections 2023-10-30T12:32:30.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.018+0000 7f3372973640 1 -- 172.21.15.3:0/1855972561 wait complete. 2023-10-30T12:32:30.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.018+0000 7f3372973640 10 monclient: build_initial_monmap 2023-10-30T12:32:30.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.018+0000 7f3372973640 10 monclient: monmap: 2023-10-30T12:32:30.038 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:30.038 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:30.038 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:30.039 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:30.039 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:30.039 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:30.039 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:30.039 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:30.040 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:30.040 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:30.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.019+0000 7f3372973640 1 librados: starting msgr at 2023-10-30T12:32:30.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.019+0000 7f3372973640 1 librados: starting objecter 2023-10-30T12:32:30.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.019+0000 7f3372973640 1 Processor -- start 2023-10-30T12:32:30.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.019+0000 7f3372973640 1 -- start start 2023-10-30T12:32:30.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.019+0000 7f3372973640 1 librados: setting wanted keys 2023-10-30T12:32:30.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.019+0000 7f3372973640 1 librados: calling monclient init 2023-10-30T12:32:30.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.019+0000 7f3372973640 10 monclient: init 2023-10-30T12:32:30.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:30.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:32:30.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f336c0b1b00 0x7f336c1da3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:30.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 10 monclient(hunting): picked mon.noname-b con 0x7f336c0b1b00 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:30.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f336c0b4390 0x7f336c1353e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:30.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 10 monclient(hunting): picked mon.noname-c con 0x7f336c0b4390 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:30.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f336c1418a0 0x7f336c135920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:30.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 10 monclient(hunting): picked mon.noname-a con 0x7f336c1418a0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:30.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:30.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c142480 con 0x7f336c1418a0 2023-10-30T12:32:30.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:30.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c0b55a0 con 0x7f336c0b1b00 2023-10-30T12:32:30.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:30.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f336c145bf0 con 0x7f336c0b4390 2023-10-30T12:32:30.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:30.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3372973640 10 monclient(hunting): authenticate will time out at 2741.234289s 2023-10-30T12:32:30.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f336bfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f336c0b1b00 0x7f336c1da3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:30.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f3370f10640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f336c1418a0 0x7f336c135920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:30.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.020+0000 7f336b7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f336c0b4390 0x7f336c1353e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:30.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336bfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f336c0b1b00 0x7f336c1da3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:33220/0 (socket says 172.21.15.3:33220) 2023-10-30T12:32:30.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336bfff640 1 -- 172.21.15.3:0/3019177397 learned_addr learned my addr 172.21.15.3:0/3019177397 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:30.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336bfff640 10 monclient(hunting): get_auth_request con 0x7f336c0b1b00 auth_method 0 2023-10-30T12:32:30.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:30.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:30.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:30.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f3370f10640 10 monclient(hunting): get_auth_request con 0x7f336c1418a0 auth_method 0 2023-10-30T12:32:30.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f3370f10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:30.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f3370f10640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:30.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f3370f10640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:30.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336b7fe640 10 monclient(hunting): get_auth_request con 0x7f336c0b4390 auth_method 0 2023-10-30T12:32:30.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:30.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:30.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:30.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:30.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:30.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:30.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f3370f10640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:30.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f3370f10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:30.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f3370f10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:30.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:30.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:30.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.021+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:30.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.024+0000 7f336bfff640 10 monclient(hunting): handle_auth_done global_id 21793 payload 824 2023-10-30T12:32:30.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.024+0000 7f336bfff640 1 -- 172.21.15.3:0/3019177397 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f336c0b4390 msgr2=0x7f336c1353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:30.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.024+0000 7f336bfff640 1 --2- 172.21.15.3:0/3019177397 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f336c0b4390 0x7f336c1353e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.024+0000 7f336bfff640 1 -- 172.21.15.3:0/3019177397 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f336c1418a0 msgr2=0x7f336c135920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:30.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.024+0000 7f336bfff640 1 --2- 172.21.15.3:0/3019177397 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f336c1418a0 0x7f336c135920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.024+0000 7f336bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:32:30.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.024+0000 7f336bfff640 1 monclient: found mon.noname-b 2023-10-30T12:32:30.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.024+0000 7f336bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:30.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.024+0000 7f336bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:30.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.024+0000 7f336bfff640 1 -- 172.21.15.3:0/3019177397 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c135e60 con 0x7f336c0b1b00 2023-10-30T12:32:30.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.026+0000 7f336bfff640 1 --2- 172.21.15.3:0/3019177397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f336c0b1b00 0x7f336c1da3e0 secure :-1 s=READY pgs=5298 cs=0 l=1 rev1=1 crypto rx=0x7f335c00e9b0 tx=0x7f335c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:30.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.026+0000 7f3368ff9640 10 client.?.objecter ms_handle_connect 0x7f336c0b1b00 2023-10-30T12:32:30.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.026+0000 7f3368ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:30.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3368ff9640 1 -- 172.21.15.3:0/3019177397 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c00c9a0 con 0x7f336c0b1b00 2023-10-30T12:32:30.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3368ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:30.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3368ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:30.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3368ff9640 10 monclient: dump: 2023-10-30T12:32:30.054 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:30.054 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:30.055 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.055 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.055 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:30.055 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:30.055 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:30.056 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:30.056 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:30.056 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:30.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3368ff9640 10 monclient: _finish_auth 0 2023-10-30T12:32:30.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3368ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:32:30.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3368ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:30.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3368ff9640 1 -- 172.21.15.3:0/3019177397 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335c00d040 con 0x7f336c0b1b00 2023-10-30T12:32:30.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3368ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:30.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3368ff9640 1 -- 172.21.15.3:0/3019177397 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c00cd50 con 0x7f336c0b1b00 2023-10-30T12:32:30.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3368ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:30.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3368ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:30.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3368ff9640 10 monclient: dump: 2023-10-30T12:32:30.058 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:30.058 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:30.058 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.058 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.059 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:30.059 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:30.059 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:30.059 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:30.060 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:30.060 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:30.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3372973640 5 monclient: authenticate success, global_id 21793 2023-10-30T12:32:30.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3372973640 10 monclient: _renew_subs 2023-10-30T12:32:30.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3372973640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:30.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3372973640 1 -- 172.21.15.3:0/3019177397 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336c143770 con 0x7f336c0b1b00 2023-10-30T12:32:30.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f336a7fc640 4 set_mon_vals no callback set 2023-10-30T12:32:30.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3372973640 10 client.21793.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:30.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3372973640 10 monclient: _renew_subs 2023-10-30T12:32:30.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3372973640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:30.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3372973640 1 -- 172.21.15.3:0/3019177397 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336c1dae00 con 0x7f336c0b1b00 2023-10-30T12:32:30.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.029+0000 7f3372973640 1 librados: init done 2023-10-30T12:32:30.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.030+0000 7f3372973640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:30.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.030+0000 7f3372973640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:30.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.030+0000 7f3372973640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:30.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.030+0000 7f3372973640 1 -- 172.21.15.3:0/3019177397 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336c0a8f60 con 0x7f336c0b1b00 2023-10-30T12:32:30.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.270+0000 7f3368ff9640 1 -- 172.21.15.3:0/3019177397 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f335c013470 con 0x7f336c0b1b00 2023-10-30T12:32:30.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.270+0000 7f3368ff9640 1 --2- 172.21.15.3:0/3019177397 >> v2:172.21.15.92:6800/1750155735 conn(0x7f335806c840 0x7f335806ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:30.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.270+0000 7f336b7fe640 1 --2- 172.21.15.3:0/3019177397 >> v2:172.21.15.92:6800/1750155735 conn(0x7f335806c840 0x7f335806ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:30.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.270+0000 7f336b7fe640 10 monclient: get_auth_request con 0x7f335806c840 auth_method 0 2023-10-30T12:32:30.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.271+0000 7f336b7fe640 1 --2- 172.21.15.3:0/3019177397 >> v2:172.21.15.92:6800/1750155735 conn(0x7f335806c840 0x7f335806ed00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f336c13c520 tx=0x7f3360009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:30.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.271+0000 7f3368ff9640 10 client.21793.objecter ms_handle_connect 0x7f335806c840 2023-10-30T12:32:30.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.272+0000 7f3368ff9640 1 -- 172.21.15.3:0/3019177397 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(899..899 src has 627..899) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f335c058790 con 0x7f336c0b1b00 2023-10-30T12:32:30.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.272+0000 7f3368ff9640 10 client.21793.objecter ms_dispatch 0x7f336c000b90 osd_map(899..899 src has 627..899) v4 2023-10-30T12:32:30.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.272+0000 7f3368ff9640 3 client.21793.objecter handle_osd_map got epochs [899,899] > 0 2023-10-30T12:32:30.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.272+0000 7f3368ff9640 3 client.21793.objecter handle_osd_map decoding full epoch 899 2023-10-30T12:32:30.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.272+0000 7f3368ff9640 20 client.21793.objecter dump_active .. 0 homeless 2023-10-30T12:32:30.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.286+0000 7f3368ff9640 1 -- 172.21.15.3:0/3019177397 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f335c05c830 con 0x7f336c0b1b00 2023-10-30T12:32:30.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.286+0000 7f3368ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:30.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.286+0000 7f3368ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:32:30.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.458+0000 7f3372973640 1 -- 172.21.15.3:0/3019177397 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f336c0b56f0 con 0x7f335806c840 2023-10-30T12:32:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.463+0000 7f3368ff9640 1 -- 172.21.15.3:0/3019177397 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151604 (secure 0 0 0) 0x7f336c0b56f0 con 0x7f335806c840 2023-10-30T12:32:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:32:30.467 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-10-30T12:32:30.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 10 librados: watch_flush enter 2023-10-30T12:32:30.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 10 librados: watch_flush exit 2023-10-30T12:32:30.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 20 client.21793.objecter shutdown clearing up homeless session... 2023-10-30T12:32:30.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 10 client.21793.objecter successfully canceled tick 2023-10-30T12:32:30.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 1 -- 172.21.15.3:0/3019177397 >> v2:172.21.15.92:6800/1750155735 conn(0x7f335806c840 msgr2=0x7f335806ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:30.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 1 --2- 172.21.15.3:0/3019177397 >> v2:172.21.15.92:6800/1750155735 conn(0x7f335806c840 0x7f335806ed00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f336c13c520 tx=0x7f3360009190 comp rx=0 tx=0).stop 2023-10-30T12:32:30.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 10 monclient: shutdown 2023-10-30T12:32:30.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:30.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 1 -- 172.21.15.3:0/3019177397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f336c0b1b00 msgr2=0x7f336c1da3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:30.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 1 --2- 172.21.15.3:0/3019177397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f336c0b1b00 0x7f336c1da3e0 secure :-1 s=READY pgs=5298 cs=0 l=1 rev1=1 crypto rx=0x7f335c00e9b0 tx=0x7f335c00ee70 comp rx=0 tx=0).stop 2023-10-30T12:32:30.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 1 -- 172.21.15.3:0/3019177397 shutdown_connections 2023-10-30T12:32:30.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 1 --2- 172.21.15.3:0/3019177397 >> v2:172.21.15.92:6800/1750155735 conn(0x7f335806c840 0x7f335806ed00 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 1 --2- 172.21.15.3:0/3019177397 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f336c1418a0 0x7f336c135920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 1 --2- 172.21.15.3:0/3019177397 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f336c0b4390 0x7f336c1353e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 1 --2- 172.21.15.3:0/3019177397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f336c0b1b00 0x7f336c1da3e0 unknown :-1 s=CLOSED pgs=5298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 1 -- 172.21.15.3:0/3019177397 >> 172.21.15.3:0/3019177397 conn(0x7f336c065ea0 msgr2=0x7f336c13ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:30.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 1 -- 172.21.15.3:0/3019177397 shutdown_connections 2023-10-30T12:32:30.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 1 -- 172.21.15.3:0/3019177397 wait complete. 2023-10-30T12:32:30.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.466+0000 7f3372973640 1 librados: shutdown 2023-10-30T12:32:30.478 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1577,"stamp":"2023-10-30T12:32:30.106095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":581480,"kb_used_data":55456,"kb_used_omap":12,"kb_used_meta":525491,"kb_avail":754393240,"statfs":{"total":773094113280,"available":772498677760,"internally_reserved":0,"allocated":56786944,"data_stored":27756542,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12743,"internal_metadata":538103353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":489,"apply_latency_ms":489,"commit_latency_ns":489000000,"apply_latency_ns":489000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.061490"},"pg_stats":[{"pgid":"24.4","version":"0'0","reported_seq":852,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.438891+0000","last_change":"2023-10-30T12:11:50.816793+0000","last_active":"2023-10-30T12:32:27.438891+0000","last_peered":"2023-10-30T12:32:27.438891+0000","last_clean":"2023-10-30T12:32:27.438891+0000","last_became_active":"2023-10-30T12:11:50.811294+0000","last_became_peered":"2023-10-30T12:11:50.811294+0000","last_unstale":"2023-10-30T12:32:27.438891+0000","last_undegraded":"2023-10-30T12:32:27.438891+0000","last_fullsized":"2023-10-30T12:32:27.438891+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:34:55.305107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":882,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.133692+0000","last_change":"2023-10-30T12:11:58.574669+0000","last_active":"2023-10-30T12:32:26.133692+0000","last_peered":"2023-10-30T12:32:26.133692+0000","last_clean":"2023-10-30T12:32:26.133692+0000","last_became_active":"2023-10-30T12:11:58.558569+0000","last_became_peered":"2023-10-30T12:11:58.558569+0000","last_unstale":"2023-10-30T12:32:26.133692+0000","last_undegraded":"2023-10-30T12:32:26.133692+0000","last_fullsized":"2023-10-30T12:32:26.133692+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:17:50.972635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":882,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.187530+0000","last_change":"2023-10-30T12:11:58.500558+0000","last_active":"2023-10-30T12:32:26.187530+0000","last_peered":"2023-10-30T12:32:26.187530+0000","last_clean":"2023-10-30T12:32:26.187530+0000","last_became_active":"2023-10-30T12:11:58.495252+0000","last_became_peered":"2023-10-30T12:11:58.495252+0000","last_unstale":"2023-10-30T12:32:26.187530+0000","last_undegraded":"2023-10-30T12:32:26.187530+0000","last_fullsized":"2023-10-30T12:32:26.187530+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:27.257928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":867,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.538426+0000","last_change":"2023-10-30T12:12:25.951506+0000","last_active":"2023-10-30T12:32:27.538426+0000","last_peered":"2023-10-30T12:32:27.538426+0000","last_clean":"2023-10-30T12:32:27.538426+0000","last_became_active":"2023-10-30T12:12:25.941872+0000","last_became_peered":"2023-10-30T12:12:25.941872+0000","last_unstale":"2023-10-30T12:32:27.538426+0000","last_undegraded":"2023-10-30T12:32:27.538426+0000","last_fullsized":"2023-10-30T12:32:27.538426+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:34.170866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":868,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.118078+0000","last_change":"2023-10-30T12:11:51.588272+0000","last_active":"2023-10-30T12:32:26.118078+0000","last_peered":"2023-10-30T12:32:26.118078+0000","last_clean":"2023-10-30T12:32:26.118078+0000","last_became_active":"2023-10-30T12:11:51.581699+0000","last_became_peered":"2023-10-30T12:11:51.581699+0000","last_unstale":"2023-10-30T12:32:26.118078+0000","last_undegraded":"2023-10-30T12:32:26.118078+0000","last_fullsized":"2023-10-30T12:32:26.118078+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:08.811631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1","version":"50'7","reported_seq":866,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.756412+0000","last_change":"2023-10-30T12:12:02.329571+0000","last_active":"2023-10-30T12:32:27.756412+0000","last_peered":"2023-10-30T12:32:27.756412+0000","last_clean":"2023-10-30T12:32:27.756412+0000","last_became_active":"2023-10-30T12:12:02.320185+0000","last_became_peered":"2023-10-30T12:12:02.320185+0000","last_unstale":"2023-10-30T12:32:27.756412+0000","last_undegraded":"2023-10-30T12:32:27.756412+0000","last_fullsized":"2023-10-30T12:32:27.756412+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:19:24.782158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":853,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.809068+0000","last_change":"2023-10-30T12:12:02.278839+0000","last_active":"2023-10-30T12:32:27.809068+0000","last_peered":"2023-10-30T12:32:27.809068+0000","last_clean":"2023-10-30T12:32:27.809068+0000","last_became_active":"2023-10-30T12:12:02.269307+0000","last_became_peered":"2023-10-30T12:12:02.269307+0000","last_unstale":"2023-10-30T12:32:27.809068+0000","last_undegraded":"2023-10-30T12:32:27.809068+0000","last_fullsized":"2023-10-30T12:32:27.809068+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:34.516157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":853,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.764967+0000","last_change":"2023-10-30T12:11:55.586707+0000","last_active":"2023-10-30T12:32:27.764967+0000","last_peered":"2023-10-30T12:32:27.764967+0000","last_clean":"2023-10-30T12:32:27.764967+0000","last_became_active":"2023-10-30T12:11:55.581296+0000","last_became_peered":"2023-10-30T12:11:55.581296+0000","last_unstale":"2023-10-30T12:32:27.764967+0000","last_undegraded":"2023-10-30T12:32:27.764967+0000","last_fullsized":"2023-10-30T12:32:27.764967+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:04.280909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":867,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.512145+0000","last_change":"2023-10-30T12:12:25.845763+0000","last_active":"2023-10-30T12:32:27.512145+0000","last_peered":"2023-10-30T12:32:27.512145+0000","last_clean":"2023-10-30T12:32:27.512145+0000","last_became_active":"2023-10-30T12:12:25.833774+0000","last_became_peered":"2023-10-30T12:12:25.833774+0000","last_unstale":"2023-10-30T12:32:27.512145+0000","last_undegraded":"2023-10-30T12:32:27.512145+0000","last_fullsized":"2023-10-30T12:32:27.512145+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:17.229351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":867,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.522357+0000","last_change":"2023-10-30T12:12:09.181066+0000","last_active":"2023-10-30T12:32:27.522357+0000","last_peered":"2023-10-30T12:32:27.522357+0000","last_clean":"2023-10-30T12:32:27.522357+0000","last_became_active":"2023-10-30T12:12:09.094001+0000","last_became_peered":"2023-10-30T12:12:09.094001+0000","last_unstale":"2023-10-30T12:32:27.522357+0000","last_undegraded":"2023-10-30T12:32:27.522357+0000","last_fullsized":"2023-10-30T12:32:27.522357+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:36.491842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":867,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.192648+0000","last_change":"2023-10-30T12:12:04.088614+0000","last_active":"2023-10-30T12:32:26.192648+0000","last_peered":"2023-10-30T12:32:26.192648+0000","last_clean":"2023-10-30T12:32:26.192648+0000","last_became_active":"2023-10-30T12:12:04.077824+0000","last_became_peered":"2023-10-30T12:12:04.077824+0000","last_unstale":"2023-10-30T12:32:26.192648+0000","last_undegraded":"2023-10-30T12:32:26.192648+0000","last_fullsized":"2023-10-30T12:32:26.192648+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:34.462067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":850,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.511157+0000","last_change":"2023-10-30T12:11:50.993004+0000","last_active":"2023-10-30T12:32:27.511157+0000","last_peered":"2023-10-30T12:32:27.511157+0000","last_clean":"2023-10-30T12:32:27.511157+0000","last_became_active":"2023-10-30T12:11:50.987299+0000","last_became_peered":"2023-10-30T12:11:50.987299+0000","last_unstale":"2023-10-30T12:32:27.511157+0000","last_undegraded":"2023-10-30T12:32:27.511157+0000","last_fullsized":"2023-10-30T12:32:27.511157+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:02:34.008030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":851,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.542115+0000","last_change":"2023-10-30T12:11:51.028381+0000","last_active":"2023-10-30T12:32:27.542115+0000","last_peered":"2023-10-30T12:32:27.542115+0000","last_clean":"2023-10-30T12:32:27.542115+0000","last_became_active":"2023-10-30T12:11:51.023075+0000","last_became_peered":"2023-10-30T12:11:51.023075+0000","last_unstale":"2023-10-30T12:32:27.542115+0000","last_undegraded":"2023-10-30T12:32:27.542115+0000","last_fullsized":"2023-10-30T12:32:27.542115+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:23:10.299251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":874,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.535544+0000","last_change":"2023-10-30T12:10:17.745135+0000","last_active":"2023-10-30T12:32:27.535544+0000","last_peered":"2023-10-30T12:32:27.535544+0000","last_clean":"2023-10-30T12:32:27.535544+0000","last_became_active":"2023-10-30T12:10:17.738241+0000","last_became_peered":"2023-10-30T12:10:17.738241+0000","last_unstale":"2023-10-30T12:32:27.535544+0000","last_undegraded":"2023-10-30T12:32:27.535544+0000","last_fullsized":"2023-10-30T12:32:27.535544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:01.788933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":859,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:32:24.541067+0000","last_change":"2023-10-30T12:11:52.264262+0000","last_active":"2023-10-30T12:32:24.541067+0000","last_peered":"2023-10-30T12:32:24.541067+0000","last_clean":"2023-10-30T12:32:24.541067+0000","last_became_active":"2023-10-30T12:11:52.257573+0000","last_became_peered":"2023-10-30T12:11:52.257573+0000","last_unstale":"2023-10-30T12:32:24.541067+0000","last_undegraded":"2023-10-30T12:32:24.541067+0000","last_fullsized":"2023-10-30T12:32:24.541067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:48:44.255249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":874,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.195697+0000","last_change":"2023-10-30T12:10:17.790588+0000","last_active":"2023-10-30T12:32:26.195697+0000","last_peered":"2023-10-30T12:32:26.195697+0000","last_clean":"2023-10-30T12:32:26.195697+0000","last_became_active":"2023-10-30T12:10:17.763227+0000","last_became_peered":"2023-10-30T12:10:17.763227+0000","last_unstale":"2023-10-30T12:32:26.195697+0000","last_undegraded":"2023-10-30T12:32:26.195697+0000","last_fullsized":"2023-10-30T12:32:26.195697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:40:26.455087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":882,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.191460+0000","last_change":"2023-10-30T12:11:55.621845+0000","last_active":"2023-10-30T12:32:26.191460+0000","last_peered":"2023-10-30T12:32:26.191460+0000","last_clean":"2023-10-30T12:32:26.191460+0000","last_became_active":"2023-10-30T12:11:55.613173+0000","last_became_peered":"2023-10-30T12:11:55.613173+0000","last_unstale":"2023-10-30T12:32:26.191460+0000","last_undegraded":"2023-10-30T12:32:26.191460+0000","last_fullsized":"2023-10-30T12:32:26.191460+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:41.422061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":874,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.520551+0000","last_change":"2023-10-30T12:10:16.912669+0000","last_active":"2023-10-30T12:32:27.520551+0000","last_peered":"2023-10-30T12:32:27.520551+0000","last_clean":"2023-10-30T12:32:27.520551+0000","last_became_active":"2023-10-30T12:10:16.907205+0000","last_became_peered":"2023-10-30T12:10:16.907205+0000","last_unstale":"2023-10-30T12:32:27.520551+0000","last_undegraded":"2023-10-30T12:32:27.520551+0000","last_fullsized":"2023-10-30T12:32:27.520551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:48.591409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":860,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:32:24.564812+0000","last_change":"2023-10-30T12:11:53.046581+0000","last_active":"2023-10-30T12:32:24.564812+0000","last_peered":"2023-10-30T12:32:24.564812+0000","last_clean":"2023-10-30T12:32:24.564812+0000","last_became_active":"2023-10-30T12:11:52.909795+0000","last_became_peered":"2023-10-30T12:11:52.909795+0000","last_unstale":"2023-10-30T12:32:24.564812+0000","last_undegraded":"2023-10-30T12:32:24.564812+0000","last_fullsized":"2023-10-30T12:32:24.564812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:40.643512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":889,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.163092+0000","last_change":"2023-10-30T12:10:17.613569+0000","last_active":"2023-10-30T12:32:26.163092+0000","last_peered":"2023-10-30T12:32:26.163092+0000","last_clean":"2023-10-30T12:32:26.163092+0000","last_became_active":"2023-10-30T12:10:17.582475+0000","last_became_peered":"2023-10-30T12:10:17.582475+0000","last_unstale":"2023-10-30T12:32:26.163092+0000","last_undegraded":"2023-10-30T12:32:26.163092+0000","last_fullsized":"2023-10-30T12:32:26.163092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:56.738579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":856,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:32:24.549332+0000","last_change":"2023-10-30T12:11:51.597894+0000","last_active":"2023-10-30T12:32:24.549332+0000","last_peered":"2023-10-30T12:32:24.549332+0000","last_clean":"2023-10-30T12:32:24.549332+0000","last_became_active":"2023-10-30T12:11:51.581905+0000","last_became_peered":"2023-10-30T12:11:51.581905+0000","last_unstale":"2023-10-30T12:32:24.549332+0000","last_undegraded":"2023-10-30T12:32:24.549332+0000","last_fullsized":"2023-10-30T12:32:24.549332+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:04.401220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":874,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.506492+0000","last_change":"2023-10-30T12:10:17.562627+0000","last_active":"2023-10-30T12:32:27.506492+0000","last_peered":"2023-10-30T12:32:27.506492+0000","last_clean":"2023-10-30T12:32:27.506492+0000","last_became_active":"2023-10-30T12:10:17.553523+0000","last_became_peered":"2023-10-30T12:10:17.553523+0000","last_unstale":"2023-10-30T12:32:27.506492+0000","last_undegraded":"2023-10-30T12:32:27.506492+0000","last_fullsized":"2023-10-30T12:32:27.506492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:05:16.522019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":851,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.556962+0000","last_change":"2023-10-30T12:11:50.971305+0000","last_active":"2023-10-30T12:32:27.556962+0000","last_peered":"2023-10-30T12:32:27.556962+0000","last_clean":"2023-10-30T12:32:27.556962+0000","last_became_active":"2023-10-30T12:11:50.965758+0000","last_became_peered":"2023-10-30T12:11:50.965758+0000","last_unstale":"2023-10-30T12:32:27.556962+0000","last_undegraded":"2023-10-30T12:32:27.556962+0000","last_fullsized":"2023-10-30T12:32:27.556962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:57:57.631020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":859,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.459754+0000","last_change":"2023-10-30T12:10:16.841521+0000","last_active":"2023-10-30T12:32:27.459754+0000","last_peered":"2023-10-30T12:32:27.459754+0000","last_clean":"2023-10-30T12:32:27.459754+0000","last_became_active":"2023-10-30T12:10:16.811711+0000","last_became_peered":"2023-10-30T12:10:16.811711+0000","last_unstale":"2023-10-30T12:32:27.459754+0000","last_undegraded":"2023-10-30T12:32:27.459754+0000","last_fullsized":"2023-10-30T12:32:27.459754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:57.059004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":851,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.422152+0000","last_change":"2023-10-30T12:12:02.414741+0000","last_active":"2023-10-30T12:32:27.422152+0000","last_peered":"2023-10-30T12:32:27.422152+0000","last_clean":"2023-10-30T12:32:27.422152+0000","last_became_active":"2023-10-30T12:12:02.408646+0000","last_became_peered":"2023-10-30T12:12:02.408646+0000","last_unstale":"2023-10-30T12:32:27.422152+0000","last_undegraded":"2023-10-30T12:32:27.422152+0000","last_fullsized":"2023-10-30T12:32:27.422152+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:48:04.995634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:32:28.659341+0000","last_change":"2023-10-30T12:10:16.879470+0000","last_active":"2023-10-30T12:32:28.659341+0000","last_peered":"2023-10-30T12:32:28.659341+0000","last_clean":"2023-10-30T12:32:28.659341+0000","last_became_active":"2023-10-30T12:10:16.849940+0000","last_became_peered":"2023-10-30T12:10:16.849940+0000","last_unstale":"2023-10-30T12:32:28.659341+0000","last_undegraded":"2023-10-30T12:32:28.659341+0000","last_fullsized":"2023-10-30T12:32:28.659341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:09:07.690364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":859,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:32:24.544761+0000","last_change":"2023-10-30T12:11:53.127463+0000","last_active":"2023-10-30T12:32:24.544761+0000","last_peered":"2023-10-30T12:32:24.544761+0000","last_clean":"2023-10-30T12:32:24.544761+0000","last_became_active":"2023-10-30T12:11:53.066917+0000","last_became_peered":"2023-10-30T12:11:53.066917+0000","last_unstale":"2023-10-30T12:32:24.544761+0000","last_undegraded":"2023-10-30T12:32:24.544761+0000","last_fullsized":"2023-10-30T12:32:24.544761+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:38.309311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":874,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.509645+0000","last_change":"2023-10-30T12:10:17.161654+0000","last_active":"2023-10-30T12:32:27.509645+0000","last_peered":"2023-10-30T12:32:27.509645+0000","last_clean":"2023-10-30T12:32:27.509645+0000","last_became_active":"2023-10-30T12:10:17.153120+0000","last_became_peered":"2023-10-30T12:10:17.153120+0000","last_unstale":"2023-10-30T12:32:27.509645+0000","last_undegraded":"2023-10-30T12:32:27.509645+0000","last_fullsized":"2023-10-30T12:32:27.509645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:42.496061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":967,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.742941+0000","last_change":"2023-10-30T12:10:09.781745+0000","last_active":"2023-10-30T12:32:27.742941+0000","last_peered":"2023-10-30T12:32:27.742941+0000","last_clean":"2023-10-30T12:32:27.742941+0000","last_became_active":"2023-10-30T12:10:09.748373+0000","last_became_peered":"2023-10-30T12:10:09.748373+0000","last_unstale":"2023-10-30T12:32:27.742941+0000","last_undegraded":"2023-10-30T12:32:27.742941+0000","last_fullsized":"2023-10-30T12:32:27.742941+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:07.000545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:07.000545+0000","last_clean_scrub_stamp":"2023-10-30T12:10:07.000545+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:30.943478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":859,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:32:24.532879+0000","last_change":"2023-10-30T12:12:03.636874+0000","last_active":"2023-10-30T12:32:24.532879+0000","last_peered":"2023-10-30T12:32:24.532879+0000","last_clean":"2023-10-30T12:32:24.532879+0000","last_became_active":"2023-10-30T12:12:03.631410+0000","last_became_peered":"2023-10-30T12:12:03.631410+0000","last_unstale":"2023-10-30T12:32:24.532879+0000","last_undegraded":"2023-10-30T12:32:24.532879+0000","last_fullsized":"2023-10-30T12:32:24.532879+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:40.711737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":868,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.211102+0000","last_change":"2023-10-30T12:12:01.935080+0000","last_active":"2023-10-30T12:32:26.211102+0000","last_peered":"2023-10-30T12:32:26.211102+0000","last_clean":"2023-10-30T12:32:26.211102+0000","last_became_active":"2023-10-30T12:12:01.926913+0000","last_became_peered":"2023-10-30T12:12:01.926913+0000","last_unstale":"2023-10-30T12:32:26.211102+0000","last_undegraded":"2023-10-30T12:32:26.211102+0000","last_fullsized":"2023-10-30T12:32:26.211102+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:08:28.436909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":851,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.456392+0000","last_change":"2023-10-30T12:12:02.114160+0000","last_active":"2023-10-30T12:32:27.456392+0000","last_peered":"2023-10-30T12:32:27.456392+0000","last_clean":"2023-10-30T12:32:27.456392+0000","last_became_active":"2023-10-30T12:12:02.108733+0000","last_became_peered":"2023-10-30T12:12:02.108733+0000","last_unstale":"2023-10-30T12:32:27.456392+0000","last_undegraded":"2023-10-30T12:32:27.456392+0000","last_fullsized":"2023-10-30T12:32:27.456392+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:57.228464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":856,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:32:24.547606+0000","last_change":"2023-10-30T12:11:52.314662+0000","last_active":"2023-10-30T12:32:24.547606+0000","last_peered":"2023-10-30T12:32:24.547606+0000","last_clean":"2023-10-30T12:32:24.547606+0000","last_became_active":"2023-10-30T12:11:52.304910+0000","last_became_peered":"2023-10-30T12:11:52.304910+0000","last_unstale":"2023-10-30T12:32:24.547606+0000","last_undegraded":"2023-10-30T12:32:24.547606+0000","last_fullsized":"2023-10-30T12:32:24.547606+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:25:49.066874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":866,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.532728+0000","last_change":"2023-10-30T12:12:10.797605+0000","last_active":"2023-10-30T12:32:27.532728+0000","last_peered":"2023-10-30T12:32:27.532728+0000","last_clean":"2023-10-30T12:32:27.532728+0000","last_became_active":"2023-10-30T12:12:10.791950+0000","last_became_peered":"2023-10-30T12:12:10.791950+0000","last_unstale":"2023-10-30T12:32:27.532728+0000","last_undegraded":"2023-10-30T12:32:27.532728+0000","last_fullsized":"2023-10-30T12:32:27.532728+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:18.345944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":868,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.229662+0000","last_change":"2023-10-30T12:11:56.058505+0000","last_active":"2023-10-30T12:32:26.229662+0000","last_peered":"2023-10-30T12:32:26.229662+0000","last_clean":"2023-10-30T12:32:26.229662+0000","last_became_active":"2023-10-30T12:11:56.049790+0000","last_became_peered":"2023-10-30T12:11:56.049790+0000","last_unstale":"2023-10-30T12:32:26.229662+0000","last_undegraded":"2023-10-30T12:32:26.229662+0000","last_fullsized":"2023-10-30T12:32:26.229662+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:10.872206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":851,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.538587+0000","last_change":"2023-10-30T12:12:02.443381+0000","last_active":"2023-10-30T12:32:27.538587+0000","last_peered":"2023-10-30T12:32:27.538587+0000","last_clean":"2023-10-30T12:32:27.538587+0000","last_became_active":"2023-10-30T12:12:02.435431+0000","last_became_peered":"2023-10-30T12:12:02.435431+0000","last_unstale":"2023-10-30T12:32:27.538587+0000","last_undegraded":"2023-10-30T12:32:27.538587+0000","last_fullsized":"2023-10-30T12:32:27.538587+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:04.786017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":851,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.441507+0000","last_change":"2023-10-30T12:11:51.298662+0000","last_active":"2023-10-30T12:32:27.441507+0000","last_peered":"2023-10-30T12:32:27.441507+0000","last_clean":"2023-10-30T12:32:27.441507+0000","last_became_active":"2023-10-30T12:11:51.289446+0000","last_became_peered":"2023-10-30T12:11:51.289446+0000","last_unstale":"2023-10-30T12:32:27.441507+0000","last_undegraded":"2023-10-30T12:32:27.441507+0000","last_fullsized":"2023-10-30T12:32:27.441507+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:03:40.836674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":852,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.559760+0000","last_change":"2023-10-30T12:11:53.122439+0000","last_active":"2023-10-30T12:32:27.559760+0000","last_peered":"2023-10-30T12:32:27.559760+0000","last_clean":"2023-10-30T12:32:27.559760+0000","last_became_active":"2023-10-30T12:11:53.110390+0000","last_became_peered":"2023-10-30T12:11:53.110390+0000","last_unstale":"2023-10-30T12:32:27.559760+0000","last_undegraded":"2023-10-30T12:32:27.559760+0000","last_fullsized":"2023-10-30T12:32:27.559760+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:52.930583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":867,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.133886+0000","last_change":"2023-10-30T12:12:00.804118+0000","last_active":"2023-10-30T12:32:26.133886+0000","last_peered":"2023-10-30T12:32:26.133886+0000","last_clean":"2023-10-30T12:32:26.133886+0000","last_became_active":"2023-10-30T12:12:00.784060+0000","last_became_peered":"2023-10-30T12:12:00.784060+0000","last_unstale":"2023-10-30T12:32:26.133886+0000","last_undegraded":"2023-10-30T12:32:26.133886+0000","last_fullsized":"2023-10-30T12:32:26.133886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:58:44.332584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":867,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.131879+0000","last_change":"2023-10-30T12:12:01.684690+0000","last_active":"2023-10-30T12:32:26.131879+0000","last_peered":"2023-10-30T12:32:26.131879+0000","last_clean":"2023-10-30T12:32:26.131879+0000","last_became_active":"2023-10-30T12:12:01.679359+0000","last_became_peered":"2023-10-30T12:12:01.679359+0000","last_unstale":"2023-10-30T12:32:26.131879+0000","last_undegraded":"2023-10-30T12:32:26.131879+0000","last_fullsized":"2023-10-30T12:32:26.131879+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:14.805349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":867,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.500626+0000","last_change":"2023-10-30T12:12:08.288834+0000","last_active":"2023-10-30T12:32:27.500626+0000","last_peered":"2023-10-30T12:32:27.500626+0000","last_clean":"2023-10-30T12:32:27.500626+0000","last_became_active":"2023-10-30T12:12:08.259922+0000","last_became_peered":"2023-10-30T12:12:08.259922+0000","last_unstale":"2023-10-30T12:32:27.500626+0000","last_undegraded":"2023-10-30T12:32:27.500626+0000","last_fullsized":"2023-10-30T12:32:27.500626+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:47:09.939780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477007,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78412,"kb_used_data":6972,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293428,"statfs":{"total":96636764160,"available":96556470272,"internally_reserved":0,"allocated":7139328,"data_stored":3397051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73136584},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":54,"apply_latency_ms":54,"commit_latency_ns":54000000,"apply_latency_ns":54000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:55 2023","interfaces":[{"interface":"back","average":{"1min":10.798999999999999,"5min":17.038,"15min":20.966000000000001},"min":{"1min":6.4589999999999996,"5min":4.1319999999999997,"15min":4.1319999999999997},"max":{"1min":26.298999999999999,"5min":140.75200000000001,"15min":213.76499999999999},"last":8.3000000000000007},{"interface":"front","average":{"1min":13.866,"5min":19.09,"15min":24.885999999999999},"min":{"1min":5.6989999999999998,"5min":4.21,"15min":4.1360000000000001},"max":{"1min":26.111999999999998,"5min":142.81200000000001,"15min":265.09100000000001},"last":12.805}]},{"osd":1,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":16.192,"5min":19.501000000000001,"15min":21.971},"min":{"1min":6.9690000000000003,"5min":5.3449999999999998,"15min":4.6609999999999996},"max":{"1min":60.024999999999999,"5min":119.535,"15min":164.755},"last":20.433},{"interface":"front","average":{"1min":16.702999999999999,"5min":19.878,"15min":25.541},"min":{"1min":7.5990000000000002,"5min":6.6399999999999997,"15min":6.0990000000000002},"max":{"1min":72.597999999999999,"5min":146.874,"15min":260.19099999999997},"last":26.492999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:31:55 2023","interfaces":[{"interface":"back","average":{"1min":18.702000000000002,"5min":23.050999999999998,"15min":32.445},"min":{"1min":9.2129999999999992,"5min":6.4589999999999996,"15min":6.4589999999999996},"max":{"1min":98.338999999999999,"5min":215.48400000000001,"15min":277.137},"last":11.539999999999999},{"interface":"front","average":{"1min":17.099,"5min":23.308,"15min":32.445},"min":{"1min":9.9890000000000008,"5min":6.5860000000000003,"15min":6.5860000000000003},"max":{"1min":74.525999999999996,"5min":254.81999999999999,"15min":265.447},"last":15.162000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:31:55 2023","interfaces":[{"interface":"back","average":{"1min":14.472,"5min":22.044,"15min":30.475999999999999},"min":{"1min":8.7599999999999998,"5min":8.7599999999999998,"15min":7.8780000000000001},"max":{"1min":23.257999999999999,"5min":139.517,"15min":287.06200000000001},"last":17.015000000000001},{"interface":"front","average":{"1min":17.981999999999999,"5min":24.353000000000002,"15min":34.198999999999998},"min":{"1min":11.537000000000001,"5min":11.537000000000001,"15min":10.587},"max":{"1min":28.649000000000001,"5min":130.221,"15min":259.279},"last":19.367000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":20.829000000000001,"5min":28.876999999999999,"15min":35.593000000000004},"min":{"1min":12.195,"5min":11.795,"15min":8.2550000000000008},"max":{"1min":74.197999999999993,"5min":324.26600000000002,"15min":324.26600000000002},"last":18.381},{"interface":"front","average":{"1min":15.435,"5min":19.82,"15min":24.088000000000001},"min":{"1min":7.1639999999999997,"5min":6.6900000000000004,"15min":6.5679999999999996},"max":{"1min":70.387,"5min":251.57900000000001,"15min":251.57900000000001},"last":9.9039999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":21.481999999999999,"5min":28.734000000000002,"15min":28.196999999999999},"min":{"1min":12.811999999999999,"5min":10.584,"15min":7.9370000000000003},"max":{"1min":68.438999999999993,"5min":135.005,"15min":218.80600000000001},"last":21.663},{"interface":"front","average":{"1min":13.765000000000001,"5min":20.492000000000001,"15min":27.355},"min":{"1min":9.7479999999999993,"5min":8.0120000000000005,"15min":6.8170000000000002},"max":{"1min":25.957999999999998,"5min":133.999,"15min":239.80500000000001},"last":13.911}]},{"osd":7,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":22.760999999999999,"5min":33.521999999999998,"15min":40.765000000000001},"min":{"1min":13.420999999999999,"5min":13.420999999999999,"15min":10.132999999999999},"max":{"1min":67.742999999999995,"5min":323.58199999999999,"15min":1077.606},"last":22.648},{"interface":"front","average":{"1min":15.550000000000001,"5min":26.276,"15min":29.468},"min":{"1min":9.5069999999999997,"5min":9.0050000000000008,"15min":8.5879999999999992},"max":{"1min":21.239000000000001,"5min":253.04599999999999,"15min":253.04599999999999},"last":16.309000000000001}]}]},{"osd":2,"up_from":16,"seq":68719477006,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60344,"kb_used_data":6948,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311496,"statfs":{"total":96636764160,"available":96574971904,"internally_reserved":0,"allocated":7114752,"data_stored":3385335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":76,"apply_latency_ms":76,"commit_latency_ns":76000000,"apply_latency_ns":76000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":22.933,"5min":16.960999999999999,"15min":24.920000000000002},"min":{"1min":7.641,"5min":4.5149999999999997,"15min":4.5149999999999997},"max":{"1min":205.785,"5min":205.785,"15min":398.01499999999999},"last":10.486000000000001},{"interface":"front","average":{"1min":19.140000000000001,"5min":18.173999999999999,"15min":24.411000000000001},"min":{"1min":6.8419999999999996,"5min":5.9969999999999999,"15min":4.4790000000000001},"max":{"1min":135.32499999999999,"5min":135.32499999999999,"15min":376.49299999999999},"last":8.0210000000000008}]},{"osd":1,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":23.818000000000001,"5min":19.024000000000001,"15min":26.135000000000002},"min":{"1min":7.2519999999999998,"5min":7.2519999999999998,"15min":6.9660000000000002},"max":{"1min":172.09999999999999,"5min":172.09999999999999,"15min":317.92099999999999},"last":12.393000000000001},{"interface":"front","average":{"1min":28.353999999999999,"5min":23.835000000000001,"15min":32.107999999999997},"min":{"1min":10.314,"5min":9.7509999999999994,"15min":7.9989999999999997},"max":{"1min":183.803,"5min":183.803,"15min":355.99400000000003},"last":13.548999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":29.100999999999999,"5min":23.119,"15min":29.891999999999999},"min":{"1min":11.067,"5min":9.3870000000000005,"15min":6.4080000000000004},"max":{"1min":207.35300000000001,"5min":207.35300000000001,"15min":412.42899999999997},"last":16.09},{"interface":"front","average":{"1min":20.431000000000001,"5min":17.634,"15min":28.106999999999999},"min":{"1min":6.1959999999999997,"5min":6.1959999999999997,"15min":6.1959999999999997},"max":{"1min":147.63900000000001,"5min":147.63900000000001,"15min":328.90899999999999},"last":7.3929999999999998}]},{"osd":4,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":17.556999999999999,"5min":16.536999999999999,"15min":22.847000000000001},"min":{"1min":8.3580000000000005,"5min":7.556,"15min":6.5439999999999996},"max":{"1min":44.722000000000001,"5min":69.641000000000005,"15min":326.565},"last":200.41300000000001},{"interface":"front","average":{"1min":20.829000000000001,"5min":20.02,"15min":27.216999999999999},"min":{"1min":12.151,"5min":10.289,"15min":9.8710000000000004},"max":{"1min":49.927,"5min":83.744,"15min":302.76499999999999},"last":45.310000000000002}]},{"osd":5,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":23.760999999999999,"5min":19.529,"15min":22.626000000000001},"min":{"1min":11.141,"5min":5.4139999999999997,"15min":4.9539999999999997},"max":{"1min":91.162000000000006,"5min":112.432,"15min":246.011},"last":211.773},{"interface":"front","average":{"1min":13.92,"5min":17.667999999999999,"15min":30.521999999999998},"min":{"1min":7.008,"5min":6.492,"15min":6.492},"max":{"1min":89.206000000000003,"5min":155.58699999999999,"15min":350.80399999999997},"last":208.166}]},{"osd":6,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":20.018000000000001,"5min":21.427,"15min":29.437999999999999},"min":{"1min":10.337999999999999,"5min":9.5139999999999993,"15min":7.8949999999999996},"max":{"1min":47.087000000000003,"5min":94.465000000000003,"15min":374.16699999999997},"last":170.02799999999999},{"interface":"front","average":{"1min":26.401,"5min":24.434000000000001,"15min":31.552},"min":{"1min":15.363,"5min":10.032,"15min":8.2400000000000002},"max":{"1min":82.998000000000005,"5min":98.968999999999994,"15min":333.37299999999999},"last":35.454000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":21.491,"5min":15.154,"15min":23.888000000000002},"min":{"1min":8.4600000000000009,"5min":6.8049999999999997,"15min":6.1539999999999999},"max":{"1min":88.486999999999995,"5min":88.486999999999995,"15min":298.68900000000002},"last":9.7690000000000001},{"interface":"front","average":{"1min":26.132000000000001,"5min":19.689,"15min":32.052999999999997},"min":{"1min":9.7449999999999992,"5min":9.7449999999999992,"15min":7.4480000000000004},"max":{"1min":114.895,"5min":114.895,"15min":453.28399999999999},"last":11.118}]}]},{"osd":0,"up_from":16,"seq":68719477006,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60360,"kb_used_data":6972,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311480,"statfs":{"total":96636764160,"available":96574955520,"internally_reserved":0,"allocated":7139328,"data_stored":3397051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":12.099,"5min":18.317,"15min":20.573},"min":{"1min":6.7759999999999998,"5min":5.6520000000000001,"15min":4.79},"max":{"1min":29.474,"5min":152.46799999999999,"15min":212.69200000000001},"last":7.5979999999999999},{"interface":"front","average":{"1min":11.606999999999999,"5min":17.617000000000001,"15min":20.478000000000002},"min":{"1min":5.1799999999999997,"5min":5.1799999999999997,"15min":5.1799999999999997},"max":{"1min":27.994,"5min":150.90100000000001,"15min":219.46600000000001},"last":8.4109999999999996}]},{"osd":2,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":16.555,"5min":17.678000000000001,"15min":21.452999999999999},"min":{"1min":7.8090000000000002,"5min":5.9580000000000002,"15min":5.9580000000000002},"max":{"1min":80.406999999999996,"5min":120.355,"15min":294.21699999999998},"last":12.138},{"interface":"front","average":{"1min":17.379999999999999,"5min":17.045000000000002,"15min":19.501000000000001},"min":{"1min":8.4209999999999994,"5min":6.5499999999999998,"15min":5.8010000000000002},"max":{"1min":96.001999999999995,"5min":101.89100000000001,"15min":237.929},"last":13.430999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":12.368,"5min":20.166,"15min":19.789000000000001},"min":{"1min":5.484,"5min":5.484,"15min":5.484},"max":{"1min":68.653000000000006,"5min":210.88900000000001,"15min":210.88900000000001},"last":6.1289999999999996},{"interface":"front","average":{"1min":17.181000000000001,"5min":17.488,"15min":21.372},"min":{"1min":7.4690000000000003,"5min":6.2770000000000001,"15min":5.8129999999999997},"max":{"1min":72.388999999999996,"5min":220.589,"15min":220.589},"last":9.048}]},{"osd":4,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":20.411999999999999,"5min":23.530999999999999,"15min":21.559000000000001},"min":{"1min":10.009,"5min":8.8040000000000003,"15min":6.4329999999999998},"max":{"1min":110.872,"5min":247.57300000000001,"15min":247.57300000000001},"last":12.781000000000001},{"interface":"front","average":{"1min":13.563000000000001,"5min":19.027999999999999,"15min":21.359000000000002},"min":{"1min":7.7889999999999997,"5min":6.7370000000000001,"15min":6.7370000000000001},"max":{"1min":23.960999999999999,"5min":267.23899999999998,"15min":267.23899999999998},"last":9.7590000000000003}]},{"osd":5,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":13.832000000000001,"5min":21.916,"15min":21.706},"min":{"1min":7.3899999999999997,"5min":7.3899999999999997,"15min":6.7080000000000002},"max":{"1min":25.125,"5min":232.041,"15min":232.041},"last":15.644},{"interface":"front","average":{"1min":16.873000000000001,"5min":25.713000000000001,"15min":25.507000000000001},"min":{"1min":11.106,"5min":9.843,"15min":9.843},"max":{"1min":27.648,"5min":256.58699999999999,"15min":256.58699999999999},"last":16.501000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":21.965,"5min":23.777999999999999,"15min":24.338999999999999},"min":{"1min":11.445,"5min":8.2750000000000004,"15min":8.2750000000000004},"max":{"1min":116.67100000000001,"5min":136.44399999999999,"15min":235.15199999999999},"last":14.853999999999999},{"interface":"front","average":{"1min":18.707000000000001,"5min":24.120999999999999,"15min":24.725000000000001},"min":{"1min":9.9090000000000007,"5min":8.4619999999999997,"15min":8.4619999999999997},"max":{"1min":56.292000000000002,"5min":196.33799999999999,"15min":196.33799999999999},"last":18.506}]},{"osd":7,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":14.186,"5min":17.329000000000001,"15min":20.686},"min":{"1min":7.851,"5min":7.2110000000000003,"15min":7.2110000000000003},"max":{"1min":20.734000000000002,"5min":95.358999999999995,"15min":232.815},"last":19.132999999999999},{"interface":"front","average":{"1min":19.161000000000001,"5min":22.934000000000001,"15min":26.524000000000001},"min":{"1min":10.509,"5min":9.1110000000000007,"15min":8.827},"max":{"1min":28.800999999999998,"5min":104.599,"15min":199.85300000000001},"last":14.154999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542415,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60376,"kb_used_data":6792,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311464,"statfs":{"total":96636764160,"available":96574939136,"internally_reserved":0,"allocated":6955008,"data_stored":3297854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":68,"apply_latency_ms":68,"commit_latency_ns":68000000,"apply_latency_ns":68000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":9.0199999999999996,"5min":13.888999999999999,"15min":21.597000000000001},"min":{"1min":4.7370000000000001,"5min":4.1820000000000004,"15min":3.7999999999999998},"max":{"1min":16.635999999999999,"5min":175.166,"15min":410.46699999999998},"last":10.207000000000001},{"interface":"front","average":{"1min":9.7629999999999999,"5min":14.135999999999999,"15min":23.727},"min":{"1min":4.335,"5min":4.0570000000000004,"15min":4.0570000000000004},"max":{"1min":18.619,"5min":182.875,"15min":1113.894},"last":9.048}]},{"osd":1,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":12.548,"5min":14.526,"15min":22.489000000000001},"min":{"1min":5.8879999999999999,"5min":5.1840000000000002,"15min":4.5839999999999996},"max":{"1min":38.268999999999998,"5min":71.530000000000001,"15min":484.42899999999997},"last":12.529},{"interface":"front","average":{"1min":11.441000000000001,"5min":16.390000000000001,"15min":24.693999999999999},"min":{"1min":7.0030000000000001,"5min":5.1369999999999996,"15min":4.7300000000000004},"max":{"1min":25.640999999999998,"5min":105.95399999999999,"15min":344.97699999999998},"last":13.500999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":13.994,"5min":19.303999999999998,"15min":26.847000000000001},"min":{"1min":7.8949999999999996,"5min":7.2370000000000001,"15min":5.242},"max":{"1min":23.893000000000001,"5min":179.96600000000001,"15min":443.49000000000001},"last":8.157},{"interface":"front","average":{"1min":15.140000000000001,"5min":20.611000000000001,"15min":28.088000000000001},"min":{"1min":7.6820000000000004,"5min":5.883,"15min":5.883},"max":{"1min":30.905000000000001,"5min":186.96899999999999,"15min":318.04000000000002},"last":14.106}]},{"osd":3,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":12.912000000000001,"5min":22.318000000000001,"15min":28.292000000000002},"min":{"1min":7.7729999999999997,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":19.795000000000002,"5min":237.43899999999999,"15min":419.54599999999999},"last":15.699999999999999},{"interface":"front","average":{"1min":16.077999999999999,"5min":24.021000000000001,"15min":30.792999999999999},"min":{"1min":8.3559999999999999,"5min":8.0380000000000003,"15min":7.4969999999999999},"max":{"1min":43.798000000000002,"5min":280.57400000000001,"15min":353.90800000000002},"last":15.064}]},{"osd":4,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":20.606000000000002,"5min":19.699000000000002,"15min":25.931000000000001},"min":{"1min":8.0239999999999991,"5min":6.1529999999999996,"15min":5.7039999999999997},"max":{"1min":125.714,"5min":185.97200000000001,"15min":420.41399999999999},"last":11.965},{"interface":"front","average":{"1min":21.055,"5min":21.521000000000001,"15min":27.934000000000001},"min":{"1min":7.1020000000000003,"5min":6.6799999999999997,"15min":6.6799999999999997},"max":{"1min":120.68000000000001,"5min":214.81,"15min":329.49000000000001},"last":17.597999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":14.662000000000001,"5min":19.605,"15min":28.992000000000001},"min":{"1min":8.8040000000000003,"5min":8.4689999999999994,"15min":7.6449999999999996},"max":{"1min":27.105,"5min":172.79300000000001,"15min":449.233},"last":10.909000000000001},{"interface":"front","average":{"1min":20.103000000000002,"5min":24.542999999999999,"15min":33.482999999999997},"min":{"1min":11.194000000000001,"5min":11.194000000000001,"15min":10.382},"max":{"1min":47.503999999999998,"5min":173.39699999999999,"15min":347.05099999999999},"last":19.603000000000002}]},{"osd":6,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":15.154999999999999,"5min":21.318000000000001,"15min":27.535},"min":{"1min":8.4199999999999999,"5min":8.2230000000000008,"15min":6.6059999999999999},"max":{"1min":19.495000000000001,"5min":240.779,"15min":454.25400000000002},"last":17.231999999999999},{"interface":"front","average":{"1min":19.113,"5min":24.262,"15min":29.777000000000001},"min":{"1min":12.965999999999999,"5min":9.8710000000000004,"15min":9.8710000000000004},"max":{"1min":42.076000000000001,"5min":198.68199999999999,"15min":507.48700000000002},"last":18.495000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542414,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100604,"kb_used_data":6832,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94271236,"statfs":{"total":96636764160,"available":96533745664,"internally_reserved":0,"allocated":6995968,"data_stored":3695699,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":95812038},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":74,"apply_latency_ms":74,"commit_latency_ns":74000000,"apply_latency_ns":74000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":14.826000000000001,"5min":15.159000000000001,"15min":21.579999999999998},"min":{"1min":5.1609999999999996,"5min":5.1609999999999996,"15min":4.1980000000000004},"max":{"1min":64.741,"5min":92.221999999999994,"15min":431.12},"last":215.155},{"interface":"front","average":{"1min":17.555,"5min":15.711,"15min":21.766999999999999},"min":{"1min":6.2409999999999997,"5min":5.1420000000000003,"15min":4.7199999999999998},"max":{"1min":98.355999999999995,"5min":98.355999999999995,"15min":399.90300000000002},"last":192.125}]},{"osd":2,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":26.707000000000001,"5min":23.045999999999999,"15min":27.221},"min":{"1min":6.7359999999999998,"5min":6.7359999999999998,"15min":6.3339999999999996},"max":{"1min":162.07900000000001,"5min":162.07900000000001,"15min":284.32900000000001},"last":9.1750000000000007},{"interface":"front","average":{"1min":21.553000000000001,"5min":19.859000000000002,"15min":25.832000000000001},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":5.8230000000000004},"max":{"1min":118.956,"5min":118.956,"15min":331.928},"last":35.747}]},{"osd":3,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":23.033999999999999,"5min":24.411000000000001,"15min":31.416},"min":{"1min":8.1790000000000003,"5min":7.1970000000000001,"15min":6.0229999999999997},"max":{"1min":104.30500000000001,"5min":153.827,"15min":270.78100000000001},"last":21.486000000000001},{"interface":"front","average":{"1min":21.942,"5min":24.722000000000001,"15min":29.132000000000001},"min":{"1min":8.5429999999999993,"5min":6.633,"15min":6.4960000000000004},"max":{"1min":106.015,"5min":235.411,"15min":321.42700000000002},"last":10.747999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":22.050000000000001,"5min":26.515000000000001,"15min":27.753},"min":{"1min":10.422000000000001,"5min":8.4659999999999993,"15min":6.6890000000000001},"max":{"1min":143.80600000000001,"5min":196.18000000000001,"15min":278.36399999999998},"last":34.500999999999998},{"interface":"front","average":{"1min":21.295999999999999,"5min":24.879999999999999,"15min":25.135999999999999},"min":{"1min":8.8339999999999996,"5min":7.641,"15min":6.3280000000000003},"max":{"1min":148.40100000000001,"5min":219.45400000000001,"15min":284.76900000000001},"last":29.760000000000002}]},{"osd":5,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":21.213999999999999,"5min":21.710999999999999,"15min":26.812999999999999},"min":{"1min":7.077,"5min":6.3140000000000001,"15min":6.0869999999999997},"max":{"1min":117.886,"5min":127.604,"15min":346.97300000000001},"last":8.0709999999999997},{"interface":"front","average":{"1min":22.224,"5min":26.302,"15min":29.582000000000001},"min":{"1min":10.473000000000001,"5min":9.8249999999999993,"15min":8.2370000000000001},"max":{"1min":110.979,"5min":280.072,"15min":313.72899999999998},"last":22.420999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":21.413,"5min":24.916,"15min":30.504999999999999},"min":{"1min":8.9939999999999998,"5min":6.7089999999999996,"15min":6.7089999999999996},"max":{"1min":126.279,"5min":126.279,"15min":391.726},"last":46.243000000000002},{"interface":"front","average":{"1min":25.158999999999999,"5min":24.603000000000002,"15min":27.167000000000002},"min":{"1min":8.7750000000000004,"5min":8.2289999999999992,"15min":8.0549999999999997},"max":{"1min":149.857,"5min":187.96799999999999,"15min":288.44099999999997},"last":24.614000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":25.628,"5min":25.503,"15min":39.037999999999997},"min":{"1min":12.448,"5min":8.6899999999999995,"15min":8.6899999999999995},"max":{"1min":137.25299999999999,"5min":137.25299999999999,"15min":1115.931},"last":203.876},{"interface":"front","average":{"1min":29.222999999999999,"5min":32.040999999999997,"15min":39.862000000000002},"min":{"1min":13.659000000000001,"5min":10.536,"15min":9.5079999999999991},"max":{"1min":151.57300000000001,"5min":199.08099999999999,"15min":1115.2660000000001},"last":222.464}]}]},{"osd":6,"up_from":13,"seq":55834575117,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60372,"kb_used_data":6972,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311468,"statfs":{"total":96636764160,"available":96574943232,"internally_reserved":0,"allocated":7139328,"data_stored":3397051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":54,"apply_latency_ms":54,"commit_latency_ns":54000000,"apply_latency_ns":54000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:58 2023","interfaces":[{"interface":"back","average":{"1min":9.2989999999999995,"5min":14.662000000000001,"15min":15.426},"min":{"1min":4.0119999999999996,"5min":3.9380000000000002,"15min":3.9380000000000002},"max":{"1min":36.322000000000003,"5min":200.376,"15min":220.25700000000001},"last":110.438},{"interface":"front","average":{"1min":9.7750000000000004,"5min":16.427,"15min":15.846},"min":{"1min":5.4180000000000001,"5min":4.6449999999999996,"15min":4.1799999999999997},"max":{"1min":28.044,"5min":196.261,"15min":219.40000000000001},"last":90.683999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:31:51 2023","interfaces":[{"interface":"back","average":{"1min":13.768000000000001,"5min":18.015000000000001,"15min":18.302},"min":{"1min":7.6390000000000002,"5min":5.1180000000000003,"15min":4.306},"max":{"1min":44.209000000000003,"5min":215.86699999999999,"15min":216.90299999999999},"last":131.17500000000001},{"interface":"front","average":{"1min":15.76,"5min":19.614999999999998,"15min":18.779},"min":{"1min":7.0019999999999998,"5min":5.2039999999999997,"15min":4.5570000000000004},"max":{"1min":42.402000000000001,"5min":225.39699999999999,"15min":225.39699999999999},"last":143.58799999999999}]},{"osd":2,"last update":"Mon Oct 30 12:32:07 2023","interfaces":[{"interface":"back","average":{"1min":12.401,"5min":18.358000000000001,"15min":19.98},"min":{"1min":6.3170000000000002,"5min":5.4729999999999999,"15min":5.4729999999999999},"max":{"1min":93.713999999999999,"5min":245.411,"15min":245.411},"last":11.587999999999999},{"interface":"front","average":{"1min":14.782999999999999,"5min":22.184000000000001,"15min":22.800000000000001},"min":{"1min":8.9339999999999993,"5min":6.7679999999999998,"15min":6.7679999999999998},"max":{"1min":26.739999999999998,"5min":244.76400000000001,"15min":244.76400000000001},"last":15.132999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:31:58 2023","interfaces":[{"interface":"back","average":{"1min":14.455,"5min":25.152000000000001,"15min":22.244},"min":{"1min":7.1970000000000001,"5min":5.9649999999999999,"15min":5.9649999999999999},"max":{"1min":35.399999999999999,"5min":217.495,"15min":300.44600000000003},"last":6.7670000000000003},{"interface":"front","average":{"1min":17.216999999999999,"5min":26.016999999999999,"15min":23.670000000000002},"min":{"1min":10.24,"5min":9.6560000000000006,"15min":8.4960000000000004},"max":{"1min":31.969000000000001,"5min":249.18100000000001,"15min":249.18100000000001},"last":10.657999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:31:51 2023","interfaces":[{"interface":"back","average":{"1min":15.673,"5min":21.34,"15min":23.218},"min":{"1min":9.9610000000000003,"5min":8.75,"15min":6.556},"max":{"1min":34.320999999999998,"5min":246.57599999999999,"15min":246.57599999999999},"last":15.901999999999999},{"interface":"front","average":{"1min":21.498000000000001,"5min":20.198,"15min":19.931000000000001},"min":{"1min":7.4119999999999999,"5min":6.9870000000000001,"15min":6.5209999999999999},"max":{"1min":78.716999999999999,"5min":233.952,"15min":233.952},"last":12.555999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:31:34 2023","interfaces":[{"interface":"back","average":{"1min":16.387,"5min":23.277000000000001,"15min":25.347999999999999},"min":{"1min":9.3940000000000001,"5min":9.3940000000000001,"15min":7.4080000000000004},"max":{"1min":24.803000000000001,"5min":267.37799999999999,"15min":348.08699999999999},"last":10},{"interface":"front","average":{"1min":16.57,"5min":20.597000000000001,"15min":23.638000000000002},"min":{"1min":10.117000000000001,"5min":7.0170000000000003,"15min":5.9089999999999998},"max":{"1min":33.296999999999997,"5min":251.63499999999999,"15min":350.89999999999998},"last":8.3149999999999995}]},{"osd":7,"last update":"Mon Oct 30 12:31:34 2023","interfaces":[{"interface":"back","average":{"1min":20.010000000000002,"5min":26.344999999999999,"15min":25.242999999999999},"min":{"1min":11.276,"5min":7.7480000000000002,"15min":5.7359999999999998},"max":{"1min":42.716999999999999,"5min":240.048,"15min":342.39400000000001},"last":142.589},{"interface":"front","average":{"1min":19.148,"5min":25.388000000000002,"15min":25.414000000000001},"min":{"1min":8.1639999999999997,"5min":7.4089999999999998,"15min":7.4089999999999998},"max":{"1min":56.218000000000004,"5min":250.70400000000001,"15min":343.60300000000001},"last":166.40100000000001}]}]},{"osd":5,"up_from":13,"seq":55834575117,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78452,"kb_used_data":7008,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293388,"statfs":{"total":96636764160,"available":96556429312,"internally_reserved":0,"allocated":7176192,"data_stored":3794891,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":13.532999999999999,"5min":15.859999999999999,"15min":16.879999999999999},"min":{"1min":4.6799999999999997,"5min":4.0190000000000001,"15min":3.9620000000000002},"max":{"1min":54.750999999999998,"5min":128.709,"15min":439.73200000000003},"last":8.7100000000000009},{"interface":"front","average":{"1min":12.602,"5min":15.343999999999999,"15min":15.458},"min":{"1min":6.1959999999999997,"5min":5.1280000000000001,"15min":4.3170000000000002},"max":{"1min":54.161000000000001,"5min":128,"15min":180.74700000000001},"last":11.015000000000001}]},{"osd":1,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":20.91,"5min":21.420000000000002,"15min":20.076000000000001},"min":{"1min":5.9880000000000004,"5min":5.8789999999999996,"15min":5.0410000000000004},"max":{"1min":73.456999999999994,"5min":213.06700000000001,"15min":235.05199999999999},"last":15.958},{"interface":"front","average":{"1min":19.178999999999998,"5min":20.559999999999999,"15min":19.687999999999999},"min":{"1min":7.2389999999999999,"5min":5.726,"15min":5.2210000000000001},"max":{"1min":84.025000000000006,"5min":187.249,"15min":420.82900000000001},"last":11.803000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":16.927,"5min":17.652999999999999,"15min":20.605},"min":{"1min":7.4199999999999999,"5min":6.4809999999999999,"15min":5.9539999999999997},"max":{"1min":61.972000000000001,"5min":76.231999999999999,"15min":219.636},"last":12.394},{"interface":"front","average":{"1min":11.673,"5min":14.134,"15min":18.402000000000001},"min":{"1min":5.3739999999999997,"5min":4.7450000000000001,"15min":4.7450000000000001},"max":{"1min":55.536000000000001,"5min":104.125,"15min":235.387},"last":6.9450000000000003}]},{"osd":3,"last update":"Mon Oct 30 12:31:59 2023","interfaces":[{"interface":"back","average":{"1min":13.568,"5min":17.634,"15min":21.957999999999998},"min":{"1min":6.3159999999999998,"5min":6.2089999999999996,"15min":5.0739999999999998},"max":{"1min":52.534999999999997,"5min":110.496,"15min":234.13999999999999},"last":13.794},{"interface":"front","average":{"1min":17.343,"5min":19.657,"15min":24.550999999999998},"min":{"1min":8.4339999999999993,"5min":6.8070000000000004,"15min":6.5330000000000004},"max":{"1min":58.048000000000002,"5min":135.05500000000001,"15min":232.31700000000001},"last":18.312999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":17.914000000000001,"5min":19.361999999999998,"15min":21.667999999999999},"min":{"1min":7.6159999999999997,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":59.853999999999999,"5min":200.67599999999999,"15min":265.50400000000002},"last":10.382},{"interface":"front","average":{"1min":17.628,"5min":18.516999999999999,"15min":21.404},"min":{"1min":6.3600000000000003,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":58.649000000000001,"5min":108.554,"15min":280.72000000000003},"last":19.423999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":21.899999999999999,"5min":20.135000000000002,"15min":23.164000000000001},"min":{"1min":11.425000000000001,"5min":8.1229999999999993,"15min":8.1229999999999993},"max":{"1min":76.626999999999995,"5min":86.918000000000006,"15min":421.87599999999998},"last":13.01},{"interface":"front","average":{"1min":20,"5min":22.122,"15min":23.684000000000001},"min":{"1min":8.8409999999999993,"5min":7.3460000000000001,"15min":7.3460000000000001},"max":{"1min":61.104999999999997,"5min":182.376,"15min":353.63099999999997},"last":20.577000000000002}]},{"osd":7,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":21.384,"5min":23.309000000000001,"15min":25.530000000000001},"min":{"1min":12.462999999999999,"5min":9.4359999999999999,"15min":6.96},"max":{"1min":59.232999999999997,"5min":124.621,"15min":854.64300000000003},"last":9.6229999999999993},{"interface":"front","average":{"1min":13.034000000000001,"5min":17.204000000000001,"15min":19.34},"min":{"1min":7.5110000000000001,"5min":6.2670000000000003,"15min":5.5300000000000002},"max":{"1min":23.940000000000001,"5min":107.976,"15min":264.84699999999998},"last":16.971}]}]},{"osd":4,"up_from":13,"seq":55834575117,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82560,"kb_used_data":6960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289280,"statfs":{"total":96636764160,"available":96552222720,"internally_reserved":0,"allocated":7127040,"data_stored":3391610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":11.680999999999999,"5min":17.094000000000001,"15min":21.251999999999999},"min":{"1min":6.5650000000000004,"5min":5.1920000000000002,"15min":3.8650000000000002},"max":{"1min":35.334000000000003,"5min":166.20400000000001,"15min":372.52300000000002},"last":118.92400000000001},{"interface":"front","average":{"1min":14.744,"5min":20.887,"15min":24.428999999999998},"min":{"1min":8.5109999999999992,"5min":6.0110000000000001,"15min":4.9660000000000002},"max":{"1min":30.943000000000001,"5min":167.251,"15min":200.37299999999999},"last":144.375}]},{"osd":1,"last update":"Mon Oct 30 12:31:31 2023","interfaces":[{"interface":"back","average":{"1min":16.497,"5min":19.984000000000002,"15min":22.559999999999999},"min":{"1min":6.9900000000000002,"5min":6.9900000000000002,"15min":5.7850000000000001},"max":{"1min":88.116,"5min":200.69300000000001,"15min":231.49799999999999},"last":126.119},{"interface":"front","average":{"1min":12.497999999999999,"5min":16.387,"15min":20.524999999999999},"min":{"1min":5.4790000000000001,"5min":4.641,"15min":4.641},"max":{"1min":86.438999999999993,"5min":223.441,"15min":223.441},"last":112.584}]},{"osd":2,"last update":"Mon Oct 30 12:31:31 2023","interfaces":[{"interface":"back","average":{"1min":20.100999999999999,"5min":26.134,"15min":28.177},"min":{"1min":9.3010000000000002,"5min":6.2149999999999999,"15min":6.2149999999999999},"max":{"1min":90.870000000000005,"5min":198.66999999999999,"15min":438.15499999999997},"last":11.478999999999999},{"interface":"front","average":{"1min":17.449000000000002,"5min":19.224,"15min":22.896000000000001},"min":{"1min":6.2409999999999997,"5min":6.2220000000000004,"15min":4.944},"max":{"1min":83.019999999999996,"5min":254.06399999999999,"15min":254.06399999999999},"last":6.976}]},{"osd":3,"last update":"Mon Oct 30 12:32:04 2023","interfaces":[{"interface":"back","average":{"1min":17.190999999999999,"5min":23.48,"15min":28.890000000000001},"min":{"1min":10.754,"5min":6.9710000000000001,"15min":6.9710000000000001},"max":{"1min":52.381999999999998,"5min":269.06700000000001,"15min":269.06700000000001},"last":12.909000000000001},{"interface":"front","average":{"1min":11.754,"5min":16.59,"15min":22.030999999999999},"min":{"1min":4.8230000000000004,"5min":4.8230000000000004,"15min":4.8230000000000004},"max":{"1min":53.374000000000002,"5min":239.715,"15min":239.715},"last":8.6170000000000009}]},{"osd":5,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":16.629000000000001,"5min":23.850000000000001,"15min":24.184999999999999},"min":{"1min":10.301,"5min":8.0670000000000002,"15min":8.0039999999999996},"max":{"1min":38.497,"5min":268.34199999999998,"15min":268.34199999999998},"last":20.835000000000001},{"interface":"front","average":{"1min":15.297000000000001,"5min":20.027000000000001,"15min":21.085000000000001},"min":{"1min":8.8490000000000002,"5min":6.1379999999999999,"15min":5.452},"max":{"1min":36.853000000000002,"5min":252.899,"15min":252.899},"last":9.4480000000000004}]},{"osd":6,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":17.149000000000001,"5min":20.524999999999999,"15min":24.902000000000001},"min":{"1min":8.9629999999999992,"5min":8.8559999999999999,"15min":7.21},"max":{"1min":51.32,"5min":273.90100000000001,"15min":273.90100000000001},"last":12.343},{"interface":"front","average":{"1min":8.6199999999999992,"5min":20.821000000000002,"15min":28.239000000000001},"min":{"1min":5.8849999999999998,"5min":5.8849999999999998,"15min":5.8849999999999998},"max":{"1min":13.861000000000001,"5min":292.11099999999999,"15min":292.11099999999999},"last":10.472}]},{"osd":7,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":17.405999999999999,"5min":25.300000000000001,"15min":30.454999999999998},"min":{"1min":11.175000000000001,"5min":7.4740000000000002,"15min":6.46},"max":{"1min":38.466000000000001,"5min":239.166,"15min":652.50699999999995},"last":185.114},{"interface":"front","average":{"1min":21.138000000000002,"5min":24.059999999999999,"15min":27.285},"min":{"1min":12.268000000000001,"5min":10.422000000000001,"15min":8.9309999999999992},"max":{"1min":114.399,"5min":303.19999999999999,"15min":303.19999999999999},"last":132.596}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:32:30.482 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T12:32:30.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:32:30.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.587+0000 7f2392644640 10 monclient: get_monmap_and_config 2023-10-30T12:32:30.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.587+0000 7f2392644640 10 monclient: build_initial_monmap 2023-10-30T12:32:30.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.588+0000 7f2392644640 10 monclient: monmap: 2023-10-30T12:32:30.590 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:30.590 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:30.590 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:30.588979+0000 2023-10-30T12:32:30.590 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:30.588979+0000 2023-10-30T12:32:30.591 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:30.591 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:30.591 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:30.591 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:30.592 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:30.592 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:30.592 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:30.592 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:30.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.588+0000 7f2392644640 1 Processor -- start 2023-10-30T12:32:30.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.588+0000 7f2392644640 1 -- start start 2023-10-30T12:32:30.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.588+0000 7f2392644640 10 monclient: init 2023-10-30T12:32:30.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.589+0000 7f2392644640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:30.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.589+0000 7f2392644640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-10-30T12:32:30.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.589+0000 7f2392644640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f238c0b4390 addr v1:172.21.15.3:6789/0 2023-10-30T12:32:30.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.589+0000 7f2392644640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f238c0b12d0 0x7f238c13f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:30.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.589+0000 7f2392644640 10 monclient(hunting): picked mon.noname-a con 0x7f238c0b12d0 addr v2:172.21.15.3:3300/0 2023-10-30T12:32:30.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.589+0000 7f2392644640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f238c141910 0x7f238c141da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:30.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.589+0000 7f2392644640 10 monclient(hunting): picked mon.noname-b con 0x7f238c141910 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:30.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.589+0000 7f2392644640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c0aa440 con 0x7f238c0b4390 2023-10-30T12:32:30.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.589+0000 7f2392644640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:30.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.589+0000 7f2392644640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f238c067a10 con 0x7f238c0b12d0 2023-10-30T12:32:30.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.589+0000 7f2392644640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:30.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.589+0000 7f2392644640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0b4770 con 0x7f238c141910 2023-10-30T12:32:30.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.589+0000 7f2392644640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:30.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.589+0000 7f2392644640 10 monclient(hunting): authenticate will time out at 2741.803515s 2023-10-30T12:32:30.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.590+0000 7f2390be1640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f238c141910 0x7f238c141da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:30.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.590+0000 7f238b7fe640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f238c0b12d0 0x7f238c13f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:30.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.590+0000 7f2390be1640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f238c141910 0x7f238c141da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:53592/0 (socket says 172.21.15.3:53592) 2023-10-30T12:32:30.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.590+0000 7f2390be1640 1 -- 172.21.15.3:0/1505113208 learned_addr learned my addr 172.21.15.3:0/1505113208 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:30.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.590+0000 7f2390be1640 10 monclient(hunting): get_auth_request con 0x7f238c141910 auth_method 0 2023-10-30T12:32:30.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.590+0000 7f2390be1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:30.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.590+0000 7f2390be1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:30.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.590+0000 7f2390be1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:30.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.590+0000 7f238b7fe640 10 monclient(hunting): get_auth_request con 0x7f238c0b12d0 auth_method 0 2023-10-30T12:32:30.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.590+0000 7f238b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:30.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.590+0000 7f238b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:30.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.590+0000 7f238b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:30.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.591+0000 7f2390be1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:30.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.591+0000 7f2390be1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:30.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.591+0000 7f2390be1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:30.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.592+0000 7f2390be1640 10 monclient(hunting): handle_auth_done global_id 15542 payload 274 2023-10-30T12:32:30.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.593+0000 7f2390be1640 1 -- 172.21.15.3:0/1505113208 >> v2:172.21.15.3:3300/0 conn(0x7f238c0b12d0 msgr2=0x7f238c13f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:30.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.593+0000 7f2390be1640 1 --2- 172.21.15.3:0/1505113208 >> v2:172.21.15.3:3300/0 conn(0x7f238c0b12d0 0x7f238c13f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.593+0000 7f2390be1640 1 -- 172.21.15.3:0/1505113208 >> v1:172.21.15.3:6789/0 conn(0x7f238c0b4390 legacy=0x7f238c13ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:30.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.593+0000 7f2390be1640 10 monclient: _finish_hunting 0 2023-10-30T12:32:30.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.593+0000 7f2390be1640 1 monclient: found mon.noname-b 2023-10-30T12:32:30.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.593+0000 7f2390be1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:30.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.593+0000 7f2390be1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:32:30.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.593+0000 7f2390be1640 1 -- 172.21.15.3:0/1505113208 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c0a8800 con 0x7f238c141910 2023-10-30T12:32:30.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.595+0000 7f2390be1640 1 --2- 172.21.15.3:0/1505113208 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f238c141910 0x7f238c141da0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f2384009e50 tx=0x7f238402bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=983dc3b81d0422e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:30.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f238affd640 1 -- 172.21.15.3:0/1505113208 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238402cb90 con 0x7f238c141910 2023-10-30T12:32:30.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f238affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:30.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f238affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:30.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f238affd640 10 monclient: dump: 2023-10-30T12:32:30.609 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:30.609 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:30.609 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.610 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.610 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:30.610 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:30.610 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:30.610 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:30.611 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:30.611 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:30.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f238affd640 10 monclient: _finish_auth 0 2023-10-30T12:32:30.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f238affd640 10 monclient: _check_auth_tickets 2023-10-30T12:32:30.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f238affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:30.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f2392644640 5 monclient: authenticate success, global_id 15542 2023-10-30T12:32:30.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f238affd640 1 -- 172.21.15.3:0/1505113208 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238402cd00 con 0x7f238c141910 2023-10-30T12:32:30.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f2392644640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:32:30.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f238affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:30.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f2392644640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:30.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f2392644640 4 set_mon_vals no callback set 2023-10-30T12:32:30.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f238affd640 1 -- 172.21.15.3:0/1505113208 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384034610 con 0x7f238c141910 2023-10-30T12:32:30.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f238affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:30.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f238affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:32:30.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f238affd640 10 monclient: dump: 2023-10-30T12:32:30.613 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:30.614 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:30.614 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.614 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.614 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:30.614 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:30.615 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:30.615 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:30.615 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:30.615 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:30.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f2392644640 10 monclient: shutdown 2023-10-30T12:32:30.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f2392644640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:30.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f2392644640 1 -- 172.21.15.3:0/1505113208 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f238c141910 msgr2=0x7f238c141da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:30.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.599+0000 7f2392644640 1 --2- 172.21.15.3:0/1505113208 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f238c141910 0x7f238c141da0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f2384009e50 tx=0x7f238402bff0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.600+0000 7f2392644640 1 -- 172.21.15.3:0/1505113208 shutdown_connections 2023-10-30T12:32:30.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.600+0000 7f2392644640 1 --2- 172.21.15.3:0/1505113208 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f238c141910 0x7f238c141da0 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.600+0000 7f2392644640 1 --2- 172.21.15.3:0/1505113208 >> v2:172.21.15.3:3300/0 conn(0x7f238c0b12d0 0x7f238c13f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.600+0000 7f2392644640 1 -- 172.21.15.3:0/1505113208 >> 172.21.15.3:0/1505113208 conn(0x7f238c065ed0 msgr2=0x7f238c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:30.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.600+0000 7f2392644640 1 -- 172.21.15.3:0/1505113208 shutdown_connections 2023-10-30T12:32:30.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.600+0000 7f2392644640 1 -- 172.21.15.3:0/1505113208 wait complete. 2023-10-30T12:32:30.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.601+0000 7f2392644640 10 monclient: build_initial_monmap 2023-10-30T12:32:30.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.601+0000 7f2392644640 10 monclient: monmap: 2023-10-30T12:32:30.618 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:30.618 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:30.618 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:30.618 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:30.619 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:30.619 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:30.619 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:30.619 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:30.619 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:30.620 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:30.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.601+0000 7f2392644640 1 librados: starting msgr at 2023-10-30T12:32:30.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.601+0000 7f2392644640 1 librados: starting objecter 2023-10-30T12:32:30.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.601+0000 7f2392644640 1 Processor -- start 2023-10-30T12:32:30.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.601+0000 7f2392644640 1 -- start start 2023-10-30T12:32:30.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.601+0000 7f2392644640 1 librados: setting wanted keys 2023-10-30T12:32:30.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.601+0000 7f2392644640 1 librados: calling monclient init 2023-10-30T12:32:30.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.601+0000 7f2392644640 10 monclient: init 2023-10-30T12:32:30.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:30.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:32:30.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f238c0b12d0 0x7f238c13aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:30.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 10 monclient(hunting): picked mon.noname-b con 0x7f238c0b12d0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:30.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f238c0b4390 0x7f238c13b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:30.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 10 monclient(hunting): picked mon.noname-a con 0x7f238c0b4390 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:30.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f238c141910 0x7f238c13ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:30.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 10 monclient(hunting): picked mon.noname-c con 0x7f238c141910 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:30.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:30.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0b2e40 con 0x7f238c0b4390 2023-10-30T12:32:30.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:30.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c139060 con 0x7f238c0b12d0 2023-10-30T12:32:30.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:30.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238c13b020 con 0x7f238c141910 2023-10-30T12:32:30.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:30.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2392644640 10 monclient(hunting): authenticate will time out at 2741.816300s 2023-10-30T12:32:30.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2390be1640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f238c141910 0x7f238c13ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:30.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f238b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f238c0b4390 0x7f238c13b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:30.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2390be1640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f238c141910 0x7f238c13ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:53604/0 (socket says 172.21.15.3:53604) 2023-10-30T12:32:30.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2390be1640 1 -- 172.21.15.3:0/2489323366 learned_addr learned my addr 172.21.15.3:0/2489323366 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:30.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2390be1640 10 monclient(hunting): get_auth_request con 0x7f238c141910 auth_method 0 2023-10-30T12:32:30.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2390be1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:30.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2390be1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:30.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f2390be1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:30.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.602+0000 7f238b7fe640 10 monclient(hunting): get_auth_request con 0x7f238c0b4390 auth_method 0 2023-10-30T12:32:30.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f238b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:30.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f238b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:30.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f238b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:30.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f238bfff640 1 --2- 172.21.15.3:0/2489323366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f238c0b12d0 0x7f238c13aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:30.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f238bfff640 10 monclient(hunting): get_auth_request con 0x7f238c0b12d0 auth_method 0 2023-10-30T12:32:30.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f238bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:30.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f238bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:30.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f238bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:30.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f2390be1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:30.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f2390be1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:30.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f2390be1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:30.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f238b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:30.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f238b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:30.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f238b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:30.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f238bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:30.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f238bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:30.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.603+0000 7f238bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:30.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.606+0000 7f2390be1640 10 monclient(hunting): handle_auth_done global_id 15545 payload 824 2023-10-30T12:32:30.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.606+0000 7f2390be1640 1 -- 172.21.15.3:0/2489323366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f238c0b12d0 msgr2=0x7f238c13aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:30.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.606+0000 7f2390be1640 1 --2- 172.21.15.3:0/2489323366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f238c0b12d0 0x7f238c13aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.606+0000 7f2390be1640 1 -- 172.21.15.3:0/2489323366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f238c0b4390 msgr2=0x7f238c13b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:30.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.606+0000 7f2390be1640 1 --2- 172.21.15.3:0/2489323366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f238c0b4390 0x7f238c13b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.606+0000 7f2390be1640 10 monclient: _finish_hunting 0 2023-10-30T12:32:30.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.606+0000 7f2390be1640 1 monclient: found mon.noname-c 2023-10-30T12:32:30.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.606+0000 7f2390be1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:30.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.606+0000 7f2390be1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.3:3301/0 2023-10-30T12:32:30.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.606+0000 7f2390be1640 1 -- 172.21.15.3:0/2489323366 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c13bfa0 con 0x7f238c141910 2023-10-30T12:32:30.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.608+0000 7f2390be1640 1 --2- 172.21.15.3:0/2489323366 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f238c141910 0x7f238c13ba60 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f238402c600 tx=0x7f2384004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:30.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.608+0000 7f2388ff9640 10 client.?.objecter ms_handle_connect 0x7f238c141910 2023-10-30T12:32:30.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.608+0000 7f2388ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:30.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2388ff9640 1 -- 172.21.15.3:0/2489323366 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384009510 con 0x7f238c141910 2023-10-30T12:32:30.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2388ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:30.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2388ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:32:30.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2388ff9640 10 monclient: dump: 2023-10-30T12:32:30.640 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:30.640 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:30.642 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:30.642 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:30.642 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:30.642 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:30.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2388ff9640 10 monclient: _finish_auth 0 2023-10-30T12:32:30.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2388ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:32:30.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2388ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:30.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2388ff9640 1 -- 172.21.15.3:0/2489323366 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238402ce80 con 0x7f238c141910 2023-10-30T12:32:30.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2388ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:30.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2388ff9640 1 -- 172.21.15.3:0/2489323366 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238403e6b0 con 0x7f238c141910 2023-10-30T12:32:30.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2388ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:30.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2388ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:32:30.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2388ff9640 10 monclient: dump: 2023-10-30T12:32:30.644 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:30.645 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:30.645 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.645 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:30.645 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:30.646 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:30.646 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:30.646 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:30.646 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:30.647 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:30.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2392644640 5 monclient: authenticate success, global_id 15545 2023-10-30T12:32:30.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2392644640 10 monclient: _renew_subs 2023-10-30T12:32:30.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2392644640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:32:30.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2392644640 1 -- 172.21.15.3:0/2489323366 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f238c13dd60 con 0x7f238c141910 2023-10-30T12:32:30.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f238a7fc640 4 set_mon_vals no callback set 2023-10-30T12:32:30.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2392644640 10 client.15545.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:30.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2392644640 10 monclient: _renew_subs 2023-10-30T12:32:30.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2392644640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:32:30.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2392644640 1 -- 172.21.15.3:0/2489323366 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238c13d830 con 0x7f238c141910 2023-10-30T12:32:30.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.612+0000 7f2392644640 1 librados: init done 2023-10-30T12:32:30.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.613+0000 7f2392644640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:30.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.613+0000 7f2392644640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:30.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.613+0000 7f2392644640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:32:30.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.613+0000 7f2392644640 1 -- 172.21.15.3:0/2489323366 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238c1396d0 con 0x7f238c141910 2023-10-30T12:32:30.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.633+0000 7f2388ff9640 1 -- 172.21.15.3:0/2489323366 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f238402cba0 con 0x7f238c141910 2023-10-30T12:32:30.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.633+0000 7f2388ff9640 1 --2- 172.21.15.3:0/2489323366 >> v2:172.21.15.92:6800/1750155735 conn(0x7f237806c840 0x7f237806ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:30.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.633+0000 7f238bfff640 1 --2- 172.21.15.3:0/2489323366 >> v2:172.21.15.92:6800/1750155735 conn(0x7f237806c840 0x7f237806ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:30.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.633+0000 7f238bfff640 10 monclient: get_auth_request con 0x7f237806c840 auth_method 0 2023-10-30T12:32:30.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.634+0000 7f238bfff640 1 --2- 172.21.15.3:0/2489323366 >> v2:172.21.15.92:6800/1750155735 conn(0x7f237806c840 0x7f237806ed00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2380000f60 tx=0x7f2380009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:30.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.634+0000 7f2388ff9640 10 client.15545.objecter ms_handle_connect 0x7f237806c840 2023-10-30T12:32:30.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.889+0000 7f2388ff9640 1 -- 172.21.15.3:0/2489323366 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(899..899 src has 627..899) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f238404d030 con 0x7f238c141910 2023-10-30T12:32:30.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.889+0000 7f2388ff9640 10 client.15545.objecter ms_dispatch 0x7f238c000b90 osd_map(899..899 src has 627..899) v4 2023-10-30T12:32:30.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.889+0000 7f2388ff9640 3 client.15545.objecter handle_osd_map got epochs [899,899] > 0 2023-10-30T12:32:30.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.889+0000 7f2388ff9640 3 client.15545.objecter handle_osd_map decoding full epoch 899 2023-10-30T12:32:30.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.889+0000 7f2388ff9640 20 client.15545.objecter dump_active .. 0 homeless 2023-10-30T12:32:30.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.889+0000 7f2388ff9640 1 -- 172.21.15.3:0/2489323366 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f238407b0f0 con 0x7f238c141910 2023-10-30T12:32:30.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.889+0000 7f2388ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:30.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:30.889+0000 7f2388ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:32:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.065+0000 7f2392644640 1 -- 172.21.15.3:0/2489323366 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f238c0a9100 con 0x7f237806c840 2023-10-30T12:32:31.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.070+0000 7f2388ff9640 1 -- 172.21.15.3:0/2489323366 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151604 (secure 0 0 0) 0x7f23800014a0 con 0x7f237806c840 2023-10-30T12:32:31.072 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:32:31.073 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-10-30T12:32:31.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 10 librados: watch_flush enter 2023-10-30T12:32:31.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 10 librados: watch_flush exit 2023-10-30T12:32:31.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 20 client.15545.objecter shutdown clearing up homeless session... 2023-10-30T12:32:31.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 10 client.15545.objecter successfully canceled tick 2023-10-30T12:32:31.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 1 -- 172.21.15.3:0/2489323366 >> v2:172.21.15.92:6800/1750155735 conn(0x7f237806c840 msgr2=0x7f237806ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:31.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 1 --2- 172.21.15.3:0/2489323366 >> v2:172.21.15.92:6800/1750155735 conn(0x7f237806c840 0x7f237806ed00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2380000f60 tx=0x7f2380009040 comp rx=0 tx=0).stop 2023-10-30T12:32:31.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 10 monclient: shutdown 2023-10-30T12:32:31.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:31.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 1 -- 172.21.15.3:0/2489323366 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f238c141910 msgr2=0x7f238c13ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:31.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 1 --2- 172.21.15.3:0/2489323366 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f238c141910 0x7f238c13ba60 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f238402c600 tx=0x7f2384004a40 comp rx=0 tx=0).stop 2023-10-30T12:32:31.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 1 -- 172.21.15.3:0/2489323366 shutdown_connections 2023-10-30T12:32:31.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 1 --2- 172.21.15.3:0/2489323366 >> v2:172.21.15.92:6800/1750155735 conn(0x7f237806c840 0x7f237806ed00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:31.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 1 --2- 172.21.15.3:0/2489323366 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f238c141910 0x7f238c13ba60 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:31.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 1 --2- 172.21.15.3:0/2489323366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f238c0b4390 0x7f238c13b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:31.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 1 --2- 172.21.15.3:0/2489323366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f238c0b12d0 0x7f238c13aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:31.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.072+0000 7f2392644640 1 -- 172.21.15.3:0/2489323366 >> 172.21.15.3:0/2489323366 conn(0x7f238c065ed0 msgr2=0x7f238c1c82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:31.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.073+0000 7f2392644640 1 -- 172.21.15.3:0/2489323366 shutdown_connections 2023-10-30T12:32:31.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.073+0000 7f2392644640 1 -- 172.21.15.3:0/2489323366 wait complete. 2023-10-30T12:32:31.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.073+0000 7f2392644640 1 librados: shutdown 2023-10-30T12:32:31.085 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1577,"stamp":"2023-10-30T12:32:30.106095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":581480,"kb_used_data":55456,"kb_used_omap":12,"kb_used_meta":525491,"kb_avail":754393240,"statfs":{"total":773094113280,"available":772498677760,"internally_reserved":0,"allocated":56786944,"data_stored":27756542,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12743,"internal_metadata":538103353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":489,"apply_latency_ms":489,"commit_latency_ns":489000000,"apply_latency_ns":489000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.061490"},"pg_stats":[{"pgid":"24.4","version":"0'0","reported_seq":852,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.438891+0000","last_change":"2023-10-30T12:11:50.816793+0000","last_active":"2023-10-30T12:32:27.438891+0000","last_peered":"2023-10-30T12:32:27.438891+0000","last_clean":"2023-10-30T12:32:27.438891+0000","last_became_active":"2023-10-30T12:11:50.811294+0000","last_became_peered":"2023-10-30T12:11:50.811294+0000","last_unstale":"2023-10-30T12:32:27.438891+0000","last_undegraded":"2023-10-30T12:32:27.438891+0000","last_fullsized":"2023-10-30T12:32:27.438891+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:34:55.305107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":882,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.133692+0000","last_change":"2023-10-30T12:11:58.574669+0000","last_active":"2023-10-30T12:32:26.133692+0000","last_peered":"2023-10-30T12:32:26.133692+0000","last_clean":"2023-10-30T12:32:26.133692+0000","last_became_active":"2023-10-30T12:11:58.558569+0000","last_became_peered":"2023-10-30T12:11:58.558569+0000","last_unstale":"2023-10-30T12:32:26.133692+0000","last_undegraded":"2023-10-30T12:32:26.133692+0000","last_fullsized":"2023-10-30T12:32:26.133692+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:17:50.972635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":882,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.187530+0000","last_change":"2023-10-30T12:11:58.500558+0000","last_active":"2023-10-30T12:32:26.187530+0000","last_peered":"2023-10-30T12:32:26.187530+0000","last_clean":"2023-10-30T12:32:26.187530+0000","last_became_active":"2023-10-30T12:11:58.495252+0000","last_became_peered":"2023-10-30T12:11:58.495252+0000","last_unstale":"2023-10-30T12:32:26.187530+0000","last_undegraded":"2023-10-30T12:32:26.187530+0000","last_fullsized":"2023-10-30T12:32:26.187530+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:27.257928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":867,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.538426+0000","last_change":"2023-10-30T12:12:25.951506+0000","last_active":"2023-10-30T12:32:27.538426+0000","last_peered":"2023-10-30T12:32:27.538426+0000","last_clean":"2023-10-30T12:32:27.538426+0000","last_became_active":"2023-10-30T12:12:25.941872+0000","last_became_peered":"2023-10-30T12:12:25.941872+0000","last_unstale":"2023-10-30T12:32:27.538426+0000","last_undegraded":"2023-10-30T12:32:27.538426+0000","last_fullsized":"2023-10-30T12:32:27.538426+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:34.170866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":868,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.118078+0000","last_change":"2023-10-30T12:11:51.588272+0000","last_active":"2023-10-30T12:32:26.118078+0000","last_peered":"2023-10-30T12:32:26.118078+0000","last_clean":"2023-10-30T12:32:26.118078+0000","last_became_active":"2023-10-30T12:11:51.581699+0000","last_became_peered":"2023-10-30T12:11:51.581699+0000","last_unstale":"2023-10-30T12:32:26.118078+0000","last_undegraded":"2023-10-30T12:32:26.118078+0000","last_fullsized":"2023-10-30T12:32:26.118078+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:08.811631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1","version":"50'7","reported_seq":866,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.756412+0000","last_change":"2023-10-30T12:12:02.329571+0000","last_active":"2023-10-30T12:32:27.756412+0000","last_peered":"2023-10-30T12:32:27.756412+0000","last_clean":"2023-10-30T12:32:27.756412+0000","last_became_active":"2023-10-30T12:12:02.320185+0000","last_became_peered":"2023-10-30T12:12:02.320185+0000","last_unstale":"2023-10-30T12:32:27.756412+0000","last_undegraded":"2023-10-30T12:32:27.756412+0000","last_fullsized":"2023-10-30T12:32:27.756412+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:19:24.782158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":853,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.809068+0000","last_change":"2023-10-30T12:12:02.278839+0000","last_active":"2023-10-30T12:32:27.809068+0000","last_peered":"2023-10-30T12:32:27.809068+0000","last_clean":"2023-10-30T12:32:27.809068+0000","last_became_active":"2023-10-30T12:12:02.269307+0000","last_became_peered":"2023-10-30T12:12:02.269307+0000","last_unstale":"2023-10-30T12:32:27.809068+0000","last_undegraded":"2023-10-30T12:32:27.809068+0000","last_fullsized":"2023-10-30T12:32:27.809068+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:34.516157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":853,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.764967+0000","last_change":"2023-10-30T12:11:55.586707+0000","last_active":"2023-10-30T12:32:27.764967+0000","last_peered":"2023-10-30T12:32:27.764967+0000","last_clean":"2023-10-30T12:32:27.764967+0000","last_became_active":"2023-10-30T12:11:55.581296+0000","last_became_peered":"2023-10-30T12:11:55.581296+0000","last_unstale":"2023-10-30T12:32:27.764967+0000","last_undegraded":"2023-10-30T12:32:27.764967+0000","last_fullsized":"2023-10-30T12:32:27.764967+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:04.280909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":867,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.512145+0000","last_change":"2023-10-30T12:12:25.845763+0000","last_active":"2023-10-30T12:32:27.512145+0000","last_peered":"2023-10-30T12:32:27.512145+0000","last_clean":"2023-10-30T12:32:27.512145+0000","last_became_active":"2023-10-30T12:12:25.833774+0000","last_became_peered":"2023-10-30T12:12:25.833774+0000","last_unstale":"2023-10-30T12:32:27.512145+0000","last_undegraded":"2023-10-30T12:32:27.512145+0000","last_fullsized":"2023-10-30T12:32:27.512145+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:17.229351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":867,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.522357+0000","last_change":"2023-10-30T12:12:09.181066+0000","last_active":"2023-10-30T12:32:27.522357+0000","last_peered":"2023-10-30T12:32:27.522357+0000","last_clean":"2023-10-30T12:32:27.522357+0000","last_became_active":"2023-10-30T12:12:09.094001+0000","last_became_peered":"2023-10-30T12:12:09.094001+0000","last_unstale":"2023-10-30T12:32:27.522357+0000","last_undegraded":"2023-10-30T12:32:27.522357+0000","last_fullsized":"2023-10-30T12:32:27.522357+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:36.491842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":867,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.192648+0000","last_change":"2023-10-30T12:12:04.088614+0000","last_active":"2023-10-30T12:32:26.192648+0000","last_peered":"2023-10-30T12:32:26.192648+0000","last_clean":"2023-10-30T12:32:26.192648+0000","last_became_active":"2023-10-30T12:12:04.077824+0000","last_became_peered":"2023-10-30T12:12:04.077824+0000","last_unstale":"2023-10-30T12:32:26.192648+0000","last_undegraded":"2023-10-30T12:32:26.192648+0000","last_fullsized":"2023-10-30T12:32:26.192648+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:34.462067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":850,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.511157+0000","last_change":"2023-10-30T12:11:50.993004+0000","last_active":"2023-10-30T12:32:27.511157+0000","last_peered":"2023-10-30T12:32:27.511157+0000","last_clean":"2023-10-30T12:32:27.511157+0000","last_became_active":"2023-10-30T12:11:50.987299+0000","last_became_peered":"2023-10-30T12:11:50.987299+0000","last_unstale":"2023-10-30T12:32:27.511157+0000","last_undegraded":"2023-10-30T12:32:27.511157+0000","last_fullsized":"2023-10-30T12:32:27.511157+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:02:34.008030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":851,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.542115+0000","last_change":"2023-10-30T12:11:51.028381+0000","last_active":"2023-10-30T12:32:27.542115+0000","last_peered":"2023-10-30T12:32:27.542115+0000","last_clean":"2023-10-30T12:32:27.542115+0000","last_became_active":"2023-10-30T12:11:51.023075+0000","last_became_peered":"2023-10-30T12:11:51.023075+0000","last_unstale":"2023-10-30T12:32:27.542115+0000","last_undegraded":"2023-10-30T12:32:27.542115+0000","last_fullsized":"2023-10-30T12:32:27.542115+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:23:10.299251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":874,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.535544+0000","last_change":"2023-10-30T12:10:17.745135+0000","last_active":"2023-10-30T12:32:27.535544+0000","last_peered":"2023-10-30T12:32:27.535544+0000","last_clean":"2023-10-30T12:32:27.535544+0000","last_became_active":"2023-10-30T12:10:17.738241+0000","last_became_peered":"2023-10-30T12:10:17.738241+0000","last_unstale":"2023-10-30T12:32:27.535544+0000","last_undegraded":"2023-10-30T12:32:27.535544+0000","last_fullsized":"2023-10-30T12:32:27.535544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:01.788933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":859,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:32:24.541067+0000","last_change":"2023-10-30T12:11:52.264262+0000","last_active":"2023-10-30T12:32:24.541067+0000","last_peered":"2023-10-30T12:32:24.541067+0000","last_clean":"2023-10-30T12:32:24.541067+0000","last_became_active":"2023-10-30T12:11:52.257573+0000","last_became_peered":"2023-10-30T12:11:52.257573+0000","last_unstale":"2023-10-30T12:32:24.541067+0000","last_undegraded":"2023-10-30T12:32:24.541067+0000","last_fullsized":"2023-10-30T12:32:24.541067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:48:44.255249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":874,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.195697+0000","last_change":"2023-10-30T12:10:17.790588+0000","last_active":"2023-10-30T12:32:26.195697+0000","last_peered":"2023-10-30T12:32:26.195697+0000","last_clean":"2023-10-30T12:32:26.195697+0000","last_became_active":"2023-10-30T12:10:17.763227+0000","last_became_peered":"2023-10-30T12:10:17.763227+0000","last_unstale":"2023-10-30T12:32:26.195697+0000","last_undegraded":"2023-10-30T12:32:26.195697+0000","last_fullsized":"2023-10-30T12:32:26.195697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:40:26.455087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":882,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.191460+0000","last_change":"2023-10-30T12:11:55.621845+0000","last_active":"2023-10-30T12:32:26.191460+0000","last_peered":"2023-10-30T12:32:26.191460+0000","last_clean":"2023-10-30T12:32:26.191460+0000","last_became_active":"2023-10-30T12:11:55.613173+0000","last_became_peered":"2023-10-30T12:11:55.613173+0000","last_unstale":"2023-10-30T12:32:26.191460+0000","last_undegraded":"2023-10-30T12:32:26.191460+0000","last_fullsized":"2023-10-30T12:32:26.191460+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:41.422061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":874,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.520551+0000","last_change":"2023-10-30T12:10:16.912669+0000","last_active":"2023-10-30T12:32:27.520551+0000","last_peered":"2023-10-30T12:32:27.520551+0000","last_clean":"2023-10-30T12:32:27.520551+0000","last_became_active":"2023-10-30T12:10:16.907205+0000","last_became_peered":"2023-10-30T12:10:16.907205+0000","last_unstale":"2023-10-30T12:32:27.520551+0000","last_undegraded":"2023-10-30T12:32:27.520551+0000","last_fullsized":"2023-10-30T12:32:27.520551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:48.591409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":860,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:32:24.564812+0000","last_change":"2023-10-30T12:11:53.046581+0000","last_active":"2023-10-30T12:32:24.564812+0000","last_peered":"2023-10-30T12:32:24.564812+0000","last_clean":"2023-10-30T12:32:24.564812+0000","last_became_active":"2023-10-30T12:11:52.909795+0000","last_became_peered":"2023-10-30T12:11:52.909795+0000","last_unstale":"2023-10-30T12:32:24.564812+0000","last_undegraded":"2023-10-30T12:32:24.564812+0000","last_fullsized":"2023-10-30T12:32:24.564812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:40.643512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":889,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.163092+0000","last_change":"2023-10-30T12:10:17.613569+0000","last_active":"2023-10-30T12:32:26.163092+0000","last_peered":"2023-10-30T12:32:26.163092+0000","last_clean":"2023-10-30T12:32:26.163092+0000","last_became_active":"2023-10-30T12:10:17.582475+0000","last_became_peered":"2023-10-30T12:10:17.582475+0000","last_unstale":"2023-10-30T12:32:26.163092+0000","last_undegraded":"2023-10-30T12:32:26.163092+0000","last_fullsized":"2023-10-30T12:32:26.163092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:56.738579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":856,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:32:24.549332+0000","last_change":"2023-10-30T12:11:51.597894+0000","last_active":"2023-10-30T12:32:24.549332+0000","last_peered":"2023-10-30T12:32:24.549332+0000","last_clean":"2023-10-30T12:32:24.549332+0000","last_became_active":"2023-10-30T12:11:51.581905+0000","last_became_peered":"2023-10-30T12:11:51.581905+0000","last_unstale":"2023-10-30T12:32:24.549332+0000","last_undegraded":"2023-10-30T12:32:24.549332+0000","last_fullsized":"2023-10-30T12:32:24.549332+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:04.401220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":874,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.506492+0000","last_change":"2023-10-30T12:10:17.562627+0000","last_active":"2023-10-30T12:32:27.506492+0000","last_peered":"2023-10-30T12:32:27.506492+0000","last_clean":"2023-10-30T12:32:27.506492+0000","last_became_active":"2023-10-30T12:10:17.553523+0000","last_became_peered":"2023-10-30T12:10:17.553523+0000","last_unstale":"2023-10-30T12:32:27.506492+0000","last_undegraded":"2023-10-30T12:32:27.506492+0000","last_fullsized":"2023-10-30T12:32:27.506492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:05:16.522019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":851,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.556962+0000","last_change":"2023-10-30T12:11:50.971305+0000","last_active":"2023-10-30T12:32:27.556962+0000","last_peered":"2023-10-30T12:32:27.556962+0000","last_clean":"2023-10-30T12:32:27.556962+0000","last_became_active":"2023-10-30T12:11:50.965758+0000","last_became_peered":"2023-10-30T12:11:50.965758+0000","last_unstale":"2023-10-30T12:32:27.556962+0000","last_undegraded":"2023-10-30T12:32:27.556962+0000","last_fullsized":"2023-10-30T12:32:27.556962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:57:57.631020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":859,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.459754+0000","last_change":"2023-10-30T12:10:16.841521+0000","last_active":"2023-10-30T12:32:27.459754+0000","last_peered":"2023-10-30T12:32:27.459754+0000","last_clean":"2023-10-30T12:32:27.459754+0000","last_became_active":"2023-10-30T12:10:16.811711+0000","last_became_peered":"2023-10-30T12:10:16.811711+0000","last_unstale":"2023-10-30T12:32:27.459754+0000","last_undegraded":"2023-10-30T12:32:27.459754+0000","last_fullsized":"2023-10-30T12:32:27.459754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:57.059004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":851,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.422152+0000","last_change":"2023-10-30T12:12:02.414741+0000","last_active":"2023-10-30T12:32:27.422152+0000","last_peered":"2023-10-30T12:32:27.422152+0000","last_clean":"2023-10-30T12:32:27.422152+0000","last_became_active":"2023-10-30T12:12:02.408646+0000","last_became_peered":"2023-10-30T12:12:02.408646+0000","last_unstale":"2023-10-30T12:32:27.422152+0000","last_undegraded":"2023-10-30T12:32:27.422152+0000","last_fullsized":"2023-10-30T12:32:27.422152+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:48:04.995634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":875,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:32:28.659341+0000","last_change":"2023-10-30T12:10:16.879470+0000","last_active":"2023-10-30T12:32:28.659341+0000","last_peered":"2023-10-30T12:32:28.659341+0000","last_clean":"2023-10-30T12:32:28.659341+0000","last_became_active":"2023-10-30T12:10:16.849940+0000","last_became_peered":"2023-10-30T12:10:16.849940+0000","last_unstale":"2023-10-30T12:32:28.659341+0000","last_undegraded":"2023-10-30T12:32:28.659341+0000","last_fullsized":"2023-10-30T12:32:28.659341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:09:07.690364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":859,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:32:24.544761+0000","last_change":"2023-10-30T12:11:53.127463+0000","last_active":"2023-10-30T12:32:24.544761+0000","last_peered":"2023-10-30T12:32:24.544761+0000","last_clean":"2023-10-30T12:32:24.544761+0000","last_became_active":"2023-10-30T12:11:53.066917+0000","last_became_peered":"2023-10-30T12:11:53.066917+0000","last_unstale":"2023-10-30T12:32:24.544761+0000","last_undegraded":"2023-10-30T12:32:24.544761+0000","last_fullsized":"2023-10-30T12:32:24.544761+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:38.309311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":874,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.509645+0000","last_change":"2023-10-30T12:10:17.161654+0000","last_active":"2023-10-30T12:32:27.509645+0000","last_peered":"2023-10-30T12:32:27.509645+0000","last_clean":"2023-10-30T12:32:27.509645+0000","last_became_active":"2023-10-30T12:10:17.153120+0000","last_became_peered":"2023-10-30T12:10:17.153120+0000","last_unstale":"2023-10-30T12:32:27.509645+0000","last_undegraded":"2023-10-30T12:32:27.509645+0000","last_fullsized":"2023-10-30T12:32:27.509645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:42.496061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":967,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.742941+0000","last_change":"2023-10-30T12:10:09.781745+0000","last_active":"2023-10-30T12:32:27.742941+0000","last_peered":"2023-10-30T12:32:27.742941+0000","last_clean":"2023-10-30T12:32:27.742941+0000","last_became_active":"2023-10-30T12:10:09.748373+0000","last_became_peered":"2023-10-30T12:10:09.748373+0000","last_unstale":"2023-10-30T12:32:27.742941+0000","last_undegraded":"2023-10-30T12:32:27.742941+0000","last_fullsized":"2023-10-30T12:32:27.742941+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:07.000545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:07.000545+0000","last_clean_scrub_stamp":"2023-10-30T12:10:07.000545+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:30.943478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":859,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:32:24.532879+0000","last_change":"2023-10-30T12:12:03.636874+0000","last_active":"2023-10-30T12:32:24.532879+0000","last_peered":"2023-10-30T12:32:24.532879+0000","last_clean":"2023-10-30T12:32:24.532879+0000","last_became_active":"2023-10-30T12:12:03.631410+0000","last_became_peered":"2023-10-30T12:12:03.631410+0000","last_unstale":"2023-10-30T12:32:24.532879+0000","last_undegraded":"2023-10-30T12:32:24.532879+0000","last_fullsized":"2023-10-30T12:32:24.532879+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:40.711737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":868,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.211102+0000","last_change":"2023-10-30T12:12:01.935080+0000","last_active":"2023-10-30T12:32:26.211102+0000","last_peered":"2023-10-30T12:32:26.211102+0000","last_clean":"2023-10-30T12:32:26.211102+0000","last_became_active":"2023-10-30T12:12:01.926913+0000","last_became_peered":"2023-10-30T12:12:01.926913+0000","last_unstale":"2023-10-30T12:32:26.211102+0000","last_undegraded":"2023-10-30T12:32:26.211102+0000","last_fullsized":"2023-10-30T12:32:26.211102+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:08:28.436909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":851,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.456392+0000","last_change":"2023-10-30T12:12:02.114160+0000","last_active":"2023-10-30T12:32:27.456392+0000","last_peered":"2023-10-30T12:32:27.456392+0000","last_clean":"2023-10-30T12:32:27.456392+0000","last_became_active":"2023-10-30T12:12:02.108733+0000","last_became_peered":"2023-10-30T12:12:02.108733+0000","last_unstale":"2023-10-30T12:32:27.456392+0000","last_undegraded":"2023-10-30T12:32:27.456392+0000","last_fullsized":"2023-10-30T12:32:27.456392+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:57.228464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":856,"reported_epoch":895,"state":"active+clean","last_fresh":"2023-10-30T12:32:24.547606+0000","last_change":"2023-10-30T12:11:52.314662+0000","last_active":"2023-10-30T12:32:24.547606+0000","last_peered":"2023-10-30T12:32:24.547606+0000","last_clean":"2023-10-30T12:32:24.547606+0000","last_became_active":"2023-10-30T12:11:52.304910+0000","last_became_peered":"2023-10-30T12:11:52.304910+0000","last_unstale":"2023-10-30T12:32:24.547606+0000","last_undegraded":"2023-10-30T12:32:24.547606+0000","last_fullsized":"2023-10-30T12:32:24.547606+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:25:49.066874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":866,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.532728+0000","last_change":"2023-10-30T12:12:10.797605+0000","last_active":"2023-10-30T12:32:27.532728+0000","last_peered":"2023-10-30T12:32:27.532728+0000","last_clean":"2023-10-30T12:32:27.532728+0000","last_became_active":"2023-10-30T12:12:10.791950+0000","last_became_peered":"2023-10-30T12:12:10.791950+0000","last_unstale":"2023-10-30T12:32:27.532728+0000","last_undegraded":"2023-10-30T12:32:27.532728+0000","last_fullsized":"2023-10-30T12:32:27.532728+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:18.345944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":868,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.229662+0000","last_change":"2023-10-30T12:11:56.058505+0000","last_active":"2023-10-30T12:32:26.229662+0000","last_peered":"2023-10-30T12:32:26.229662+0000","last_clean":"2023-10-30T12:32:26.229662+0000","last_became_active":"2023-10-30T12:11:56.049790+0000","last_became_peered":"2023-10-30T12:11:56.049790+0000","last_unstale":"2023-10-30T12:32:26.229662+0000","last_undegraded":"2023-10-30T12:32:26.229662+0000","last_fullsized":"2023-10-30T12:32:26.229662+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:10.872206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":851,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.538587+0000","last_change":"2023-10-30T12:12:02.443381+0000","last_active":"2023-10-30T12:32:27.538587+0000","last_peered":"2023-10-30T12:32:27.538587+0000","last_clean":"2023-10-30T12:32:27.538587+0000","last_became_active":"2023-10-30T12:12:02.435431+0000","last_became_peered":"2023-10-30T12:12:02.435431+0000","last_unstale":"2023-10-30T12:32:27.538587+0000","last_undegraded":"2023-10-30T12:32:27.538587+0000","last_fullsized":"2023-10-30T12:32:27.538587+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:04.786017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":851,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.441507+0000","last_change":"2023-10-30T12:11:51.298662+0000","last_active":"2023-10-30T12:32:27.441507+0000","last_peered":"2023-10-30T12:32:27.441507+0000","last_clean":"2023-10-30T12:32:27.441507+0000","last_became_active":"2023-10-30T12:11:51.289446+0000","last_became_peered":"2023-10-30T12:11:51.289446+0000","last_unstale":"2023-10-30T12:32:27.441507+0000","last_undegraded":"2023-10-30T12:32:27.441507+0000","last_fullsized":"2023-10-30T12:32:27.441507+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:03:40.836674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":852,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.559760+0000","last_change":"2023-10-30T12:11:53.122439+0000","last_active":"2023-10-30T12:32:27.559760+0000","last_peered":"2023-10-30T12:32:27.559760+0000","last_clean":"2023-10-30T12:32:27.559760+0000","last_became_active":"2023-10-30T12:11:53.110390+0000","last_became_peered":"2023-10-30T12:11:53.110390+0000","last_unstale":"2023-10-30T12:32:27.559760+0000","last_undegraded":"2023-10-30T12:32:27.559760+0000","last_fullsized":"2023-10-30T12:32:27.559760+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:52.930583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":867,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.133886+0000","last_change":"2023-10-30T12:12:00.804118+0000","last_active":"2023-10-30T12:32:26.133886+0000","last_peered":"2023-10-30T12:32:26.133886+0000","last_clean":"2023-10-30T12:32:26.133886+0000","last_became_active":"2023-10-30T12:12:00.784060+0000","last_became_peered":"2023-10-30T12:12:00.784060+0000","last_unstale":"2023-10-30T12:32:26.133886+0000","last_undegraded":"2023-10-30T12:32:26.133886+0000","last_fullsized":"2023-10-30T12:32:26.133886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:58:44.332584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":867,"reported_epoch":897,"state":"active+clean","last_fresh":"2023-10-30T12:32:26.131879+0000","last_change":"2023-10-30T12:12:01.684690+0000","last_active":"2023-10-30T12:32:26.131879+0000","last_peered":"2023-10-30T12:32:26.131879+0000","last_clean":"2023-10-30T12:32:26.131879+0000","last_became_active":"2023-10-30T12:12:01.679359+0000","last_became_peered":"2023-10-30T12:12:01.679359+0000","last_unstale":"2023-10-30T12:32:26.131879+0000","last_undegraded":"2023-10-30T12:32:26.131879+0000","last_fullsized":"2023-10-30T12:32:26.131879+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:14.805349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":867,"reported_epoch":898,"state":"active+clean","last_fresh":"2023-10-30T12:32:27.500626+0000","last_change":"2023-10-30T12:12:08.288834+0000","last_active":"2023-10-30T12:32:27.500626+0000","last_peered":"2023-10-30T12:32:27.500626+0000","last_clean":"2023-10-30T12:32:27.500626+0000","last_became_active":"2023-10-30T12:12:08.259922+0000","last_became_peered":"2023-10-30T12:12:08.259922+0000","last_unstale":"2023-10-30T12:32:27.500626+0000","last_undegraded":"2023-10-30T12:32:27.500626+0000","last_fullsized":"2023-10-30T12:32:27.500626+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:47:09.939780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477007,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78412,"kb_used_data":6972,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293428,"statfs":{"total":96636764160,"available":96556470272,"internally_reserved":0,"allocated":7139328,"data_stored":3397051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73136584},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":54,"apply_latency_ms":54,"commit_latency_ns":54000000,"apply_latency_ns":54000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:55 2023","interfaces":[{"interface":"back","average":{"1min":10.798999999999999,"5min":17.038,"15min":20.966000000000001},"min":{"1min":6.4589999999999996,"5min":4.1319999999999997,"15min":4.1319999999999997},"max":{"1min":26.298999999999999,"5min":140.75200000000001,"15min":213.76499999999999},"last":8.3000000000000007},{"interface":"front","average":{"1min":13.866,"5min":19.09,"15min":24.885999999999999},"min":{"1min":5.6989999999999998,"5min":4.21,"15min":4.1360000000000001},"max":{"1min":26.111999999999998,"5min":142.81200000000001,"15min":265.09100000000001},"last":12.805}]},{"osd":1,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":16.192,"5min":19.501000000000001,"15min":21.971},"min":{"1min":6.9690000000000003,"5min":5.3449999999999998,"15min":4.6609999999999996},"max":{"1min":60.024999999999999,"5min":119.535,"15min":164.755},"last":20.433},{"interface":"front","average":{"1min":16.702999999999999,"5min":19.878,"15min":25.541},"min":{"1min":7.5990000000000002,"5min":6.6399999999999997,"15min":6.0990000000000002},"max":{"1min":72.597999999999999,"5min":146.874,"15min":260.19099999999997},"last":26.492999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:31:55 2023","interfaces":[{"interface":"back","average":{"1min":18.702000000000002,"5min":23.050999999999998,"15min":32.445},"min":{"1min":9.2129999999999992,"5min":6.4589999999999996,"15min":6.4589999999999996},"max":{"1min":98.338999999999999,"5min":215.48400000000001,"15min":277.137},"last":11.539999999999999},{"interface":"front","average":{"1min":17.099,"5min":23.308,"15min":32.445},"min":{"1min":9.9890000000000008,"5min":6.5860000000000003,"15min":6.5860000000000003},"max":{"1min":74.525999999999996,"5min":254.81999999999999,"15min":265.447},"last":15.162000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:31:55 2023","interfaces":[{"interface":"back","average":{"1min":14.472,"5min":22.044,"15min":30.475999999999999},"min":{"1min":8.7599999999999998,"5min":8.7599999999999998,"15min":7.8780000000000001},"max":{"1min":23.257999999999999,"5min":139.517,"15min":287.06200000000001},"last":17.015000000000001},{"interface":"front","average":{"1min":17.981999999999999,"5min":24.353000000000002,"15min":34.198999999999998},"min":{"1min":11.537000000000001,"5min":11.537000000000001,"15min":10.587},"max":{"1min":28.649000000000001,"5min":130.221,"15min":259.279},"last":19.367000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":20.829000000000001,"5min":28.876999999999999,"15min":35.593000000000004},"min":{"1min":12.195,"5min":11.795,"15min":8.2550000000000008},"max":{"1min":74.197999999999993,"5min":324.26600000000002,"15min":324.26600000000002},"last":18.381},{"interface":"front","average":{"1min":15.435,"5min":19.82,"15min":24.088000000000001},"min":{"1min":7.1639999999999997,"5min":6.6900000000000004,"15min":6.5679999999999996},"max":{"1min":70.387,"5min":251.57900000000001,"15min":251.57900000000001},"last":9.9039999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":21.481999999999999,"5min":28.734000000000002,"15min":28.196999999999999},"min":{"1min":12.811999999999999,"5min":10.584,"15min":7.9370000000000003},"max":{"1min":68.438999999999993,"5min":135.005,"15min":218.80600000000001},"last":21.663},{"interface":"front","average":{"1min":13.765000000000001,"5min":20.492000000000001,"15min":27.355},"min":{"1min":9.7479999999999993,"5min":8.0120000000000005,"15min":6.8170000000000002},"max":{"1min":25.957999999999998,"5min":133.999,"15min":239.80500000000001},"last":13.911}]},{"osd":7,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":22.760999999999999,"5min":33.521999999999998,"15min":40.765000000000001},"min":{"1min":13.420999999999999,"5min":13.420999999999999,"15min":10.132999999999999},"max":{"1min":67.742999999999995,"5min":323.58199999999999,"15min":1077.606},"last":22.648},{"interface":"front","average":{"1min":15.550000000000001,"5min":26.276,"15min":29.468},"min":{"1min":9.5069999999999997,"5min":9.0050000000000008,"15min":8.5879999999999992},"max":{"1min":21.239000000000001,"5min":253.04599999999999,"15min":253.04599999999999},"last":16.309000000000001}]}]},{"osd":2,"up_from":16,"seq":68719477006,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60344,"kb_used_data":6948,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311496,"statfs":{"total":96636764160,"available":96574971904,"internally_reserved":0,"allocated":7114752,"data_stored":3385335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":76,"apply_latency_ms":76,"commit_latency_ns":76000000,"apply_latency_ns":76000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":22.933,"5min":16.960999999999999,"15min":24.920000000000002},"min":{"1min":7.641,"5min":4.5149999999999997,"15min":4.5149999999999997},"max":{"1min":205.785,"5min":205.785,"15min":398.01499999999999},"last":10.486000000000001},{"interface":"front","average":{"1min":19.140000000000001,"5min":18.173999999999999,"15min":24.411000000000001},"min":{"1min":6.8419999999999996,"5min":5.9969999999999999,"15min":4.4790000000000001},"max":{"1min":135.32499999999999,"5min":135.32499999999999,"15min":376.49299999999999},"last":8.0210000000000008}]},{"osd":1,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":23.818000000000001,"5min":19.024000000000001,"15min":26.135000000000002},"min":{"1min":7.2519999999999998,"5min":7.2519999999999998,"15min":6.9660000000000002},"max":{"1min":172.09999999999999,"5min":172.09999999999999,"15min":317.92099999999999},"last":12.393000000000001},{"interface":"front","average":{"1min":28.353999999999999,"5min":23.835000000000001,"15min":32.107999999999997},"min":{"1min":10.314,"5min":9.7509999999999994,"15min":7.9989999999999997},"max":{"1min":183.803,"5min":183.803,"15min":355.99400000000003},"last":13.548999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":29.100999999999999,"5min":23.119,"15min":29.891999999999999},"min":{"1min":11.067,"5min":9.3870000000000005,"15min":6.4080000000000004},"max":{"1min":207.35300000000001,"5min":207.35300000000001,"15min":412.42899999999997},"last":16.09},{"interface":"front","average":{"1min":20.431000000000001,"5min":17.634,"15min":28.106999999999999},"min":{"1min":6.1959999999999997,"5min":6.1959999999999997,"15min":6.1959999999999997},"max":{"1min":147.63900000000001,"5min":147.63900000000001,"15min":328.90899999999999},"last":7.3929999999999998}]},{"osd":4,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":17.556999999999999,"5min":16.536999999999999,"15min":22.847000000000001},"min":{"1min":8.3580000000000005,"5min":7.556,"15min":6.5439999999999996},"max":{"1min":44.722000000000001,"5min":69.641000000000005,"15min":326.565},"last":200.41300000000001},{"interface":"front","average":{"1min":20.829000000000001,"5min":20.02,"15min":27.216999999999999},"min":{"1min":12.151,"5min":10.289,"15min":9.8710000000000004},"max":{"1min":49.927,"5min":83.744,"15min":302.76499999999999},"last":45.310000000000002}]},{"osd":5,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":23.760999999999999,"5min":19.529,"15min":22.626000000000001},"min":{"1min":11.141,"5min":5.4139999999999997,"15min":4.9539999999999997},"max":{"1min":91.162000000000006,"5min":112.432,"15min":246.011},"last":211.773},{"interface":"front","average":{"1min":13.92,"5min":17.667999999999999,"15min":30.521999999999998},"min":{"1min":7.008,"5min":6.492,"15min":6.492},"max":{"1min":89.206000000000003,"5min":155.58699999999999,"15min":350.80399999999997},"last":208.166}]},{"osd":6,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":20.018000000000001,"5min":21.427,"15min":29.437999999999999},"min":{"1min":10.337999999999999,"5min":9.5139999999999993,"15min":7.8949999999999996},"max":{"1min":47.087000000000003,"5min":94.465000000000003,"15min":374.16699999999997},"last":170.02799999999999},{"interface":"front","average":{"1min":26.401,"5min":24.434000000000001,"15min":31.552},"min":{"1min":15.363,"5min":10.032,"15min":8.2400000000000002},"max":{"1min":82.998000000000005,"5min":98.968999999999994,"15min":333.37299999999999},"last":35.454000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":21.491,"5min":15.154,"15min":23.888000000000002},"min":{"1min":8.4600000000000009,"5min":6.8049999999999997,"15min":6.1539999999999999},"max":{"1min":88.486999999999995,"5min":88.486999999999995,"15min":298.68900000000002},"last":9.7690000000000001},{"interface":"front","average":{"1min":26.132000000000001,"5min":19.689,"15min":32.052999999999997},"min":{"1min":9.7449999999999992,"5min":9.7449999999999992,"15min":7.4480000000000004},"max":{"1min":114.895,"5min":114.895,"15min":453.28399999999999},"last":11.118}]}]},{"osd":0,"up_from":16,"seq":68719477006,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60360,"kb_used_data":6972,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311480,"statfs":{"total":96636764160,"available":96574955520,"internally_reserved":0,"allocated":7139328,"data_stored":3397051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":12.099,"5min":18.317,"15min":20.573},"min":{"1min":6.7759999999999998,"5min":5.6520000000000001,"15min":4.79},"max":{"1min":29.474,"5min":152.46799999999999,"15min":212.69200000000001},"last":7.5979999999999999},{"interface":"front","average":{"1min":11.606999999999999,"5min":17.617000000000001,"15min":20.478000000000002},"min":{"1min":5.1799999999999997,"5min":5.1799999999999997,"15min":5.1799999999999997},"max":{"1min":27.994,"5min":150.90100000000001,"15min":219.46600000000001},"last":8.4109999999999996}]},{"osd":2,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":16.555,"5min":17.678000000000001,"15min":21.452999999999999},"min":{"1min":7.8090000000000002,"5min":5.9580000000000002,"15min":5.9580000000000002},"max":{"1min":80.406999999999996,"5min":120.355,"15min":294.21699999999998},"last":12.138},{"interface":"front","average":{"1min":17.379999999999999,"5min":17.045000000000002,"15min":19.501000000000001},"min":{"1min":8.4209999999999994,"5min":6.5499999999999998,"15min":5.8010000000000002},"max":{"1min":96.001999999999995,"5min":101.89100000000001,"15min":237.929},"last":13.430999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":12.368,"5min":20.166,"15min":19.789000000000001},"min":{"1min":5.484,"5min":5.484,"15min":5.484},"max":{"1min":68.653000000000006,"5min":210.88900000000001,"15min":210.88900000000001},"last":6.1289999999999996},{"interface":"front","average":{"1min":17.181000000000001,"5min":17.488,"15min":21.372},"min":{"1min":7.4690000000000003,"5min":6.2770000000000001,"15min":5.8129999999999997},"max":{"1min":72.388999999999996,"5min":220.589,"15min":220.589},"last":9.048}]},{"osd":4,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":20.411999999999999,"5min":23.530999999999999,"15min":21.559000000000001},"min":{"1min":10.009,"5min":8.8040000000000003,"15min":6.4329999999999998},"max":{"1min":110.872,"5min":247.57300000000001,"15min":247.57300000000001},"last":12.781000000000001},{"interface":"front","average":{"1min":13.563000000000001,"5min":19.027999999999999,"15min":21.359000000000002},"min":{"1min":7.7889999999999997,"5min":6.7370000000000001,"15min":6.7370000000000001},"max":{"1min":23.960999999999999,"5min":267.23899999999998,"15min":267.23899999999998},"last":9.7590000000000003}]},{"osd":5,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":13.832000000000001,"5min":21.916,"15min":21.706},"min":{"1min":7.3899999999999997,"5min":7.3899999999999997,"15min":6.7080000000000002},"max":{"1min":25.125,"5min":232.041,"15min":232.041},"last":15.644},{"interface":"front","average":{"1min":16.873000000000001,"5min":25.713000000000001,"15min":25.507000000000001},"min":{"1min":11.106,"5min":9.843,"15min":9.843},"max":{"1min":27.648,"5min":256.58699999999999,"15min":256.58699999999999},"last":16.501000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":21.965,"5min":23.777999999999999,"15min":24.338999999999999},"min":{"1min":11.445,"5min":8.2750000000000004,"15min":8.2750000000000004},"max":{"1min":116.67100000000001,"5min":136.44399999999999,"15min":235.15199999999999},"last":14.853999999999999},{"interface":"front","average":{"1min":18.707000000000001,"5min":24.120999999999999,"15min":24.725000000000001},"min":{"1min":9.9090000000000007,"5min":8.4619999999999997,"15min":8.4619999999999997},"max":{"1min":56.292000000000002,"5min":196.33799999999999,"15min":196.33799999999999},"last":18.506}]},{"osd":7,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":14.186,"5min":17.329000000000001,"15min":20.686},"min":{"1min":7.851,"5min":7.2110000000000003,"15min":7.2110000000000003},"max":{"1min":20.734000000000002,"5min":95.358999999999995,"15min":232.815},"last":19.132999999999999},{"interface":"front","average":{"1min":19.161000000000001,"5min":22.934000000000001,"15min":26.524000000000001},"min":{"1min":10.509,"5min":9.1110000000000007,"15min":8.827},"max":{"1min":28.800999999999998,"5min":104.599,"15min":199.85300000000001},"last":14.154999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542415,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60376,"kb_used_data":6792,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311464,"statfs":{"total":96636764160,"available":96574939136,"internally_reserved":0,"allocated":6955008,"data_stored":3297854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":68,"apply_latency_ms":68,"commit_latency_ns":68000000,"apply_latency_ns":68000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":9.0199999999999996,"5min":13.888999999999999,"15min":21.597000000000001},"min":{"1min":4.7370000000000001,"5min":4.1820000000000004,"15min":3.7999999999999998},"max":{"1min":16.635999999999999,"5min":175.166,"15min":410.46699999999998},"last":10.207000000000001},{"interface":"front","average":{"1min":9.7629999999999999,"5min":14.135999999999999,"15min":23.727},"min":{"1min":4.335,"5min":4.0570000000000004,"15min":4.0570000000000004},"max":{"1min":18.619,"5min":182.875,"15min":1113.894},"last":9.048}]},{"osd":1,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":12.548,"5min":14.526,"15min":22.489000000000001},"min":{"1min":5.8879999999999999,"5min":5.1840000000000002,"15min":4.5839999999999996},"max":{"1min":38.268999999999998,"5min":71.530000000000001,"15min":484.42899999999997},"last":12.529},{"interface":"front","average":{"1min":11.441000000000001,"5min":16.390000000000001,"15min":24.693999999999999},"min":{"1min":7.0030000000000001,"5min":5.1369999999999996,"15min":4.7300000000000004},"max":{"1min":25.640999999999998,"5min":105.95399999999999,"15min":344.97699999999998},"last":13.500999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":13.994,"5min":19.303999999999998,"15min":26.847000000000001},"min":{"1min":7.8949999999999996,"5min":7.2370000000000001,"15min":5.242},"max":{"1min":23.893000000000001,"5min":179.96600000000001,"15min":443.49000000000001},"last":8.157},{"interface":"front","average":{"1min":15.140000000000001,"5min":20.611000000000001,"15min":28.088000000000001},"min":{"1min":7.6820000000000004,"5min":5.883,"15min":5.883},"max":{"1min":30.905000000000001,"5min":186.96899999999999,"15min":318.04000000000002},"last":14.106}]},{"osd":3,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":12.912000000000001,"5min":22.318000000000001,"15min":28.292000000000002},"min":{"1min":7.7729999999999997,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":19.795000000000002,"5min":237.43899999999999,"15min":419.54599999999999},"last":15.699999999999999},{"interface":"front","average":{"1min":16.077999999999999,"5min":24.021000000000001,"15min":30.792999999999999},"min":{"1min":8.3559999999999999,"5min":8.0380000000000003,"15min":7.4969999999999999},"max":{"1min":43.798000000000002,"5min":280.57400000000001,"15min":353.90800000000002},"last":15.064}]},{"osd":4,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":20.606000000000002,"5min":19.699000000000002,"15min":25.931000000000001},"min":{"1min":8.0239999999999991,"5min":6.1529999999999996,"15min":5.7039999999999997},"max":{"1min":125.714,"5min":185.97200000000001,"15min":420.41399999999999},"last":11.965},{"interface":"front","average":{"1min":21.055,"5min":21.521000000000001,"15min":27.934000000000001},"min":{"1min":7.1020000000000003,"5min":6.6799999999999997,"15min":6.6799999999999997},"max":{"1min":120.68000000000001,"5min":214.81,"15min":329.49000000000001},"last":17.597999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":14.662000000000001,"5min":19.605,"15min":28.992000000000001},"min":{"1min":8.8040000000000003,"5min":8.4689999999999994,"15min":7.6449999999999996},"max":{"1min":27.105,"5min":172.79300000000001,"15min":449.233},"last":10.909000000000001},{"interface":"front","average":{"1min":20.103000000000002,"5min":24.542999999999999,"15min":33.482999999999997},"min":{"1min":11.194000000000001,"5min":11.194000000000001,"15min":10.382},"max":{"1min":47.503999999999998,"5min":173.39699999999999,"15min":347.05099999999999},"last":19.603000000000002}]},{"osd":6,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":15.154999999999999,"5min":21.318000000000001,"15min":27.535},"min":{"1min":8.4199999999999999,"5min":8.2230000000000008,"15min":6.6059999999999999},"max":{"1min":19.495000000000001,"5min":240.779,"15min":454.25400000000002},"last":17.231999999999999},{"interface":"front","average":{"1min":19.113,"5min":24.262,"15min":29.777000000000001},"min":{"1min":12.965999999999999,"5min":9.8710000000000004,"15min":9.8710000000000004},"max":{"1min":42.076000000000001,"5min":198.68199999999999,"15min":507.48700000000002},"last":18.495000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542414,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100604,"kb_used_data":6832,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94271236,"statfs":{"total":96636764160,"available":96533745664,"internally_reserved":0,"allocated":6995968,"data_stored":3695699,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":95812038},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":74,"apply_latency_ms":74,"commit_latency_ns":74000000,"apply_latency_ns":74000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":14.826000000000001,"5min":15.159000000000001,"15min":21.579999999999998},"min":{"1min":5.1609999999999996,"5min":5.1609999999999996,"15min":4.1980000000000004},"max":{"1min":64.741,"5min":92.221999999999994,"15min":431.12},"last":215.155},{"interface":"front","average":{"1min":17.555,"5min":15.711,"15min":21.766999999999999},"min":{"1min":6.2409999999999997,"5min":5.1420000000000003,"15min":4.7199999999999998},"max":{"1min":98.355999999999995,"5min":98.355999999999995,"15min":399.90300000000002},"last":192.125}]},{"osd":2,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":26.707000000000001,"5min":23.045999999999999,"15min":27.221},"min":{"1min":6.7359999999999998,"5min":6.7359999999999998,"15min":6.3339999999999996},"max":{"1min":162.07900000000001,"5min":162.07900000000001,"15min":284.32900000000001},"last":9.1750000000000007},{"interface":"front","average":{"1min":21.553000000000001,"5min":19.859000000000002,"15min":25.832000000000001},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":5.8230000000000004},"max":{"1min":118.956,"5min":118.956,"15min":331.928},"last":35.747}]},{"osd":3,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":23.033999999999999,"5min":24.411000000000001,"15min":31.416},"min":{"1min":8.1790000000000003,"5min":7.1970000000000001,"15min":6.0229999999999997},"max":{"1min":104.30500000000001,"5min":153.827,"15min":270.78100000000001},"last":21.486000000000001},{"interface":"front","average":{"1min":21.942,"5min":24.722000000000001,"15min":29.132000000000001},"min":{"1min":8.5429999999999993,"5min":6.633,"15min":6.4960000000000004},"max":{"1min":106.015,"5min":235.411,"15min":321.42700000000002},"last":10.747999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":22.050000000000001,"5min":26.515000000000001,"15min":27.753},"min":{"1min":10.422000000000001,"5min":8.4659999999999993,"15min":6.6890000000000001},"max":{"1min":143.80600000000001,"5min":196.18000000000001,"15min":278.36399999999998},"last":34.500999999999998},{"interface":"front","average":{"1min":21.295999999999999,"5min":24.879999999999999,"15min":25.135999999999999},"min":{"1min":8.8339999999999996,"5min":7.641,"15min":6.3280000000000003},"max":{"1min":148.40100000000001,"5min":219.45400000000001,"15min":284.76900000000001},"last":29.760000000000002}]},{"osd":5,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":21.213999999999999,"5min":21.710999999999999,"15min":26.812999999999999},"min":{"1min":7.077,"5min":6.3140000000000001,"15min":6.0869999999999997},"max":{"1min":117.886,"5min":127.604,"15min":346.97300000000001},"last":8.0709999999999997},{"interface":"front","average":{"1min":22.224,"5min":26.302,"15min":29.582000000000001},"min":{"1min":10.473000000000001,"5min":9.8249999999999993,"15min":8.2370000000000001},"max":{"1min":110.979,"5min":280.072,"15min":313.72899999999998},"last":22.420999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":21.413,"5min":24.916,"15min":30.504999999999999},"min":{"1min":8.9939999999999998,"5min":6.7089999999999996,"15min":6.7089999999999996},"max":{"1min":126.279,"5min":126.279,"15min":391.726},"last":46.243000000000002},{"interface":"front","average":{"1min":25.158999999999999,"5min":24.603000000000002,"15min":27.167000000000002},"min":{"1min":8.7750000000000004,"5min":8.2289999999999992,"15min":8.0549999999999997},"max":{"1min":149.857,"5min":187.96799999999999,"15min":288.44099999999997},"last":24.614000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":25.628,"5min":25.503,"15min":39.037999999999997},"min":{"1min":12.448,"5min":8.6899999999999995,"15min":8.6899999999999995},"max":{"1min":137.25299999999999,"5min":137.25299999999999,"15min":1115.931},"last":203.876},{"interface":"front","average":{"1min":29.222999999999999,"5min":32.040999999999997,"15min":39.862000000000002},"min":{"1min":13.659000000000001,"5min":10.536,"15min":9.5079999999999991},"max":{"1min":151.57300000000001,"5min":199.08099999999999,"15min":1115.2660000000001},"last":222.464}]}]},{"osd":6,"up_from":13,"seq":55834575117,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60372,"kb_used_data":6972,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311468,"statfs":{"total":96636764160,"available":96574943232,"internally_reserved":0,"allocated":7139328,"data_stored":3397051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":54,"apply_latency_ms":54,"commit_latency_ns":54000000,"apply_latency_ns":54000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:58 2023","interfaces":[{"interface":"back","average":{"1min":9.2989999999999995,"5min":14.662000000000001,"15min":15.426},"min":{"1min":4.0119999999999996,"5min":3.9380000000000002,"15min":3.9380000000000002},"max":{"1min":36.322000000000003,"5min":200.376,"15min":220.25700000000001},"last":110.438},{"interface":"front","average":{"1min":9.7750000000000004,"5min":16.427,"15min":15.846},"min":{"1min":5.4180000000000001,"5min":4.6449999999999996,"15min":4.1799999999999997},"max":{"1min":28.044,"5min":196.261,"15min":219.40000000000001},"last":90.683999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:31:51 2023","interfaces":[{"interface":"back","average":{"1min":13.768000000000001,"5min":18.015000000000001,"15min":18.302},"min":{"1min":7.6390000000000002,"5min":5.1180000000000003,"15min":4.306},"max":{"1min":44.209000000000003,"5min":215.86699999999999,"15min":216.90299999999999},"last":131.17500000000001},{"interface":"front","average":{"1min":15.76,"5min":19.614999999999998,"15min":18.779},"min":{"1min":7.0019999999999998,"5min":5.2039999999999997,"15min":4.5570000000000004},"max":{"1min":42.402000000000001,"5min":225.39699999999999,"15min":225.39699999999999},"last":143.58799999999999}]},{"osd":2,"last update":"Mon Oct 30 12:32:07 2023","interfaces":[{"interface":"back","average":{"1min":12.401,"5min":18.358000000000001,"15min":19.98},"min":{"1min":6.3170000000000002,"5min":5.4729999999999999,"15min":5.4729999999999999},"max":{"1min":93.713999999999999,"5min":245.411,"15min":245.411},"last":11.587999999999999},{"interface":"front","average":{"1min":14.782999999999999,"5min":22.184000000000001,"15min":22.800000000000001},"min":{"1min":8.9339999999999993,"5min":6.7679999999999998,"15min":6.7679999999999998},"max":{"1min":26.739999999999998,"5min":244.76400000000001,"15min":244.76400000000001},"last":15.132999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:31:58 2023","interfaces":[{"interface":"back","average":{"1min":14.455,"5min":25.152000000000001,"15min":22.244},"min":{"1min":7.1970000000000001,"5min":5.9649999999999999,"15min":5.9649999999999999},"max":{"1min":35.399999999999999,"5min":217.495,"15min":300.44600000000003},"last":6.7670000000000003},{"interface":"front","average":{"1min":17.216999999999999,"5min":26.016999999999999,"15min":23.670000000000002},"min":{"1min":10.24,"5min":9.6560000000000006,"15min":8.4960000000000004},"max":{"1min":31.969000000000001,"5min":249.18100000000001,"15min":249.18100000000001},"last":10.657999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:31:51 2023","interfaces":[{"interface":"back","average":{"1min":15.673,"5min":21.34,"15min":23.218},"min":{"1min":9.9610000000000003,"5min":8.75,"15min":6.556},"max":{"1min":34.320999999999998,"5min":246.57599999999999,"15min":246.57599999999999},"last":15.901999999999999},{"interface":"front","average":{"1min":21.498000000000001,"5min":20.198,"15min":19.931000000000001},"min":{"1min":7.4119999999999999,"5min":6.9870000000000001,"15min":6.5209999999999999},"max":{"1min":78.716999999999999,"5min":233.952,"15min":233.952},"last":12.555999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:31:34 2023","interfaces":[{"interface":"back","average":{"1min":16.387,"5min":23.277000000000001,"15min":25.347999999999999},"min":{"1min":9.3940000000000001,"5min":9.3940000000000001,"15min":7.4080000000000004},"max":{"1min":24.803000000000001,"5min":267.37799999999999,"15min":348.08699999999999},"last":10},{"interface":"front","average":{"1min":16.57,"5min":20.597000000000001,"15min":23.638000000000002},"min":{"1min":10.117000000000001,"5min":7.0170000000000003,"15min":5.9089999999999998},"max":{"1min":33.296999999999997,"5min":251.63499999999999,"15min":350.89999999999998},"last":8.3149999999999995}]},{"osd":7,"last update":"Mon Oct 30 12:31:34 2023","interfaces":[{"interface":"back","average":{"1min":20.010000000000002,"5min":26.344999999999999,"15min":25.242999999999999},"min":{"1min":11.276,"5min":7.7480000000000002,"15min":5.7359999999999998},"max":{"1min":42.716999999999999,"5min":240.048,"15min":342.39400000000001},"last":142.589},{"interface":"front","average":{"1min":19.148,"5min":25.388000000000002,"15min":25.414000000000001},"min":{"1min":8.1639999999999997,"5min":7.4089999999999998,"15min":7.4089999999999998},"max":{"1min":56.218000000000004,"5min":250.70400000000001,"15min":343.60300000000001},"last":166.40100000000001}]}]},{"osd":5,"up_from":13,"seq":55834575117,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78452,"kb_used_data":7008,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293388,"statfs":{"total":96636764160,"available":96556429312,"internally_reserved":0,"allocated":7176192,"data_stored":3794891,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":13.532999999999999,"5min":15.859999999999999,"15min":16.879999999999999},"min":{"1min":4.6799999999999997,"5min":4.0190000000000001,"15min":3.9620000000000002},"max":{"1min":54.750999999999998,"5min":128.709,"15min":439.73200000000003},"last":8.7100000000000009},{"interface":"front","average":{"1min":12.602,"5min":15.343999999999999,"15min":15.458},"min":{"1min":6.1959999999999997,"5min":5.1280000000000001,"15min":4.3170000000000002},"max":{"1min":54.161000000000001,"5min":128,"15min":180.74700000000001},"last":11.015000000000001}]},{"osd":1,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":20.91,"5min":21.420000000000002,"15min":20.076000000000001},"min":{"1min":5.9880000000000004,"5min":5.8789999999999996,"15min":5.0410000000000004},"max":{"1min":73.456999999999994,"5min":213.06700000000001,"15min":235.05199999999999},"last":15.958},{"interface":"front","average":{"1min":19.178999999999998,"5min":20.559999999999999,"15min":19.687999999999999},"min":{"1min":7.2389999999999999,"5min":5.726,"15min":5.2210000000000001},"max":{"1min":84.025000000000006,"5min":187.249,"15min":420.82900000000001},"last":11.803000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":16.927,"5min":17.652999999999999,"15min":20.605},"min":{"1min":7.4199999999999999,"5min":6.4809999999999999,"15min":5.9539999999999997},"max":{"1min":61.972000000000001,"5min":76.231999999999999,"15min":219.636},"last":12.394},{"interface":"front","average":{"1min":11.673,"5min":14.134,"15min":18.402000000000001},"min":{"1min":5.3739999999999997,"5min":4.7450000000000001,"15min":4.7450000000000001},"max":{"1min":55.536000000000001,"5min":104.125,"15min":235.387},"last":6.9450000000000003}]},{"osd":3,"last update":"Mon Oct 30 12:31:59 2023","interfaces":[{"interface":"back","average":{"1min":13.568,"5min":17.634,"15min":21.957999999999998},"min":{"1min":6.3159999999999998,"5min":6.2089999999999996,"15min":5.0739999999999998},"max":{"1min":52.534999999999997,"5min":110.496,"15min":234.13999999999999},"last":13.794},{"interface":"front","average":{"1min":17.343,"5min":19.657,"15min":24.550999999999998},"min":{"1min":8.4339999999999993,"5min":6.8070000000000004,"15min":6.5330000000000004},"max":{"1min":58.048000000000002,"5min":135.05500000000001,"15min":232.31700000000001},"last":18.312999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":17.914000000000001,"5min":19.361999999999998,"15min":21.667999999999999},"min":{"1min":7.6159999999999997,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":59.853999999999999,"5min":200.67599999999999,"15min":265.50400000000002},"last":10.382},{"interface":"front","average":{"1min":17.628,"5min":18.516999999999999,"15min":21.404},"min":{"1min":6.3600000000000003,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":58.649000000000001,"5min":108.554,"15min":280.72000000000003},"last":19.423999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":21.899999999999999,"5min":20.135000000000002,"15min":23.164000000000001},"min":{"1min":11.425000000000001,"5min":8.1229999999999993,"15min":8.1229999999999993},"max":{"1min":76.626999999999995,"5min":86.918000000000006,"15min":421.87599999999998},"last":13.01},{"interface":"front","average":{"1min":20,"5min":22.122,"15min":23.684000000000001},"min":{"1min":8.8409999999999993,"5min":7.3460000000000001,"15min":7.3460000000000001},"max":{"1min":61.104999999999997,"5min":182.376,"15min":353.63099999999997},"last":20.577000000000002}]},{"osd":7,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":21.384,"5min":23.309000000000001,"15min":25.530000000000001},"min":{"1min":12.462999999999999,"5min":9.4359999999999999,"15min":6.96},"max":{"1min":59.232999999999997,"5min":124.621,"15min":854.64300000000003},"last":9.6229999999999993},{"interface":"front","average":{"1min":13.034000000000001,"5min":17.204000000000001,"15min":19.34},"min":{"1min":7.5110000000000001,"5min":6.2670000000000003,"15min":5.5300000000000002},"max":{"1min":23.940000000000001,"5min":107.976,"15min":264.84699999999998},"last":16.971}]}]},{"osd":4,"up_from":13,"seq":55834575117,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82560,"kb_used_data":6960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289280,"statfs":{"total":96636764160,"available":96552222720,"internally_reserved":0,"allocated":7127040,"data_stored":3391610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":11.680999999999999,"5min":17.094000000000001,"15min":21.251999999999999},"min":{"1min":6.5650000000000004,"5min":5.1920000000000002,"15min":3.8650000000000002},"max":{"1min":35.334000000000003,"5min":166.20400000000001,"15min":372.52300000000002},"last":118.92400000000001},{"interface":"front","average":{"1min":14.744,"5min":20.887,"15min":24.428999999999998},"min":{"1min":8.5109999999999992,"5min":6.0110000000000001,"15min":4.9660000000000002},"max":{"1min":30.943000000000001,"5min":167.251,"15min":200.37299999999999},"last":144.375}]},{"osd":1,"last update":"Mon Oct 30 12:31:31 2023","interfaces":[{"interface":"back","average":{"1min":16.497,"5min":19.984000000000002,"15min":22.559999999999999},"min":{"1min":6.9900000000000002,"5min":6.9900000000000002,"15min":5.7850000000000001},"max":{"1min":88.116,"5min":200.69300000000001,"15min":231.49799999999999},"last":126.119},{"interface":"front","average":{"1min":12.497999999999999,"5min":16.387,"15min":20.524999999999999},"min":{"1min":5.4790000000000001,"5min":4.641,"15min":4.641},"max":{"1min":86.438999999999993,"5min":223.441,"15min":223.441},"last":112.584}]},{"osd":2,"last update":"Mon Oct 30 12:31:31 2023","interfaces":[{"interface":"back","average":{"1min":20.100999999999999,"5min":26.134,"15min":28.177},"min":{"1min":9.3010000000000002,"5min":6.2149999999999999,"15min":6.2149999999999999},"max":{"1min":90.870000000000005,"5min":198.66999999999999,"15min":438.15499999999997},"last":11.478999999999999},{"interface":"front","average":{"1min":17.449000000000002,"5min":19.224,"15min":22.896000000000001},"min":{"1min":6.2409999999999997,"5min":6.2220000000000004,"15min":4.944},"max":{"1min":83.019999999999996,"5min":254.06399999999999,"15min":254.06399999999999},"last":6.976}]},{"osd":3,"last update":"Mon Oct 30 12:32:04 2023","interfaces":[{"interface":"back","average":{"1min":17.190999999999999,"5min":23.48,"15min":28.890000000000001},"min":{"1min":10.754,"5min":6.9710000000000001,"15min":6.9710000000000001},"max":{"1min":52.381999999999998,"5min":269.06700000000001,"15min":269.06700000000001},"last":12.909000000000001},{"interface":"front","average":{"1min":11.754,"5min":16.59,"15min":22.030999999999999},"min":{"1min":4.8230000000000004,"5min":4.8230000000000004,"15min":4.8230000000000004},"max":{"1min":53.374000000000002,"5min":239.715,"15min":239.715},"last":8.6170000000000009}]},{"osd":5,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":16.629000000000001,"5min":23.850000000000001,"15min":24.184999999999999},"min":{"1min":10.301,"5min":8.0670000000000002,"15min":8.0039999999999996},"max":{"1min":38.497,"5min":268.34199999999998,"15min":268.34199999999998},"last":20.835000000000001},{"interface":"front","average":{"1min":15.297000000000001,"5min":20.027000000000001,"15min":21.085000000000001},"min":{"1min":8.8490000000000002,"5min":6.1379999999999999,"15min":5.452},"max":{"1min":36.853000000000002,"5min":252.899,"15min":252.899},"last":9.4480000000000004}]},{"osd":6,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":17.149000000000001,"5min":20.524999999999999,"15min":24.902000000000001},"min":{"1min":8.9629999999999992,"5min":8.8559999999999999,"15min":7.21},"max":{"1min":51.32,"5min":273.90100000000001,"15min":273.90100000000001},"last":12.343},{"interface":"front","average":{"1min":8.6199999999999992,"5min":20.821000000000002,"15min":28.239000000000001},"min":{"1min":5.8849999999999998,"5min":5.8849999999999998,"15min":5.8849999999999998},"max":{"1min":13.861000000000001,"5min":292.11099999999999,"15min":292.11099999999999},"last":10.472}]},{"osd":7,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":17.405999999999999,"5min":25.300000000000001,"15min":30.454999999999998},"min":{"1min":11.175000000000001,"5min":7.4740000000000002,"15min":6.46},"max":{"1min":38.466000000000001,"5min":239.166,"15min":652.50699999999995},"last":185.114},{"interface":"front","average":{"1min":21.138000000000002,"5min":24.059999999999999,"15min":27.285},"min":{"1min":12.268000000000001,"5min":10.422000000000001,"15min":8.9309999999999992},"max":{"1min":114.399,"5min":303.19999999999999,"15min":303.19999999999999},"last":132.596}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:32:31.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:32:31.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.193+0000 7f75298b4640 10 monclient: get_monmap_and_config 2023-10-30T12:32:31.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.193+0000 7f75298b4640 10 monclient: build_initial_monmap 2023-10-30T12:32:31.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.193+0000 7f75298b4640 10 monclient: monmap: 2023-10-30T12:32:31.196 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:31.196 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:31.196 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:31.194920+0000 2023-10-30T12:32:31.196 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:31.194920+0000 2023-10-30T12:32:31.196 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:31.197 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:31.197 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:31.197 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:31.197 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:31.197 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:31.197 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:31.198 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:31.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.194+0000 7f75298b4640 1 Processor -- start 2023-10-30T12:32:31.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.194+0000 7f75298b4640 1 -- start start 2023-10-30T12:32:31.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.194+0000 7f75298b4640 10 monclient: init 2023-10-30T12:32:31.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.195+0000 7f75298b4640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:31.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.195+0000 7f75298b4640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-10-30T12:32:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.195+0000 7f75298b4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7524142e40 0x7f75240abcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.195+0000 7f75298b4640 10 monclient(hunting): picked mon.noname-c con 0x7f7524142e40 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.195+0000 7f75298b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75240ac210 addr v1:172.21.15.3:6789/0 2023-10-30T12:32:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.195+0000 7f75298b4640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7524070650 0x7f7524149f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.195+0000 7f75298b4640 10 monclient(hunting): picked mon.noname-a con 0x7f7524070650 addr v2:172.21.15.3:3300/0 2023-10-30T12:32:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.195+0000 7f75298b4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7524071a90 con 0x7f75240ac210 2023-10-30T12:32:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.195+0000 7f75298b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.195+0000 7f75298b4640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7524067b90 con 0x7f7524070650 2023-10-30T12:32:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.195+0000 7f75298b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.195+0000 7f75298b4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7524070a80 con 0x7f7524142e40 2023-10-30T12:32:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.195+0000 7f75298b4640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.195+0000 7f75298b4640 10 monclient(hunting): authenticate will time out at 2742.409259s 2023-10-30T12:32:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.196+0000 7f7522ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7524142e40 0x7f75240abcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.196+0000 7f75237fe640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7524070650 0x7f7524149f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.196+0000 7f75227fc640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f75240ac210 0x7f752406ff40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52618/0 (socket says 172.21.15.3:52618) 2023-10-30T12:32:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.196+0000 7f75227fc640 1 -- 172.21.15.3:0/602019054 learned_addr learned my addr 172.21.15.3:0/602019054 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:32:31.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.196+0000 7f7522ffd640 10 monclient(hunting): get_auth_request con 0x7f7524142e40 auth_method 0 2023-10-30T12:32:31.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.196+0000 7f7522ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:31.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.196+0000 7f7522ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:31.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.196+0000 7f7522ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:31.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.196+0000 7f75237fe640 10 monclient(hunting): get_auth_request con 0x7f7524070650 auth_method 0 2023-10-30T12:32:31.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.196+0000 7f75237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:31.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.196+0000 7f75237fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:31.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.196+0000 7f75237fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:31.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.196+0000 7f7522ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:31.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.196+0000 7f7522ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:31.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.196+0000 7f7522ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:31.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.197+0000 7f75237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:31.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.197+0000 7f75237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:31.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.197+0000 7f75237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:31.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.198+0000 7f7522ffd640 10 monclient(hunting): handle_auth_done global_id 21799 payload 274 2023-10-30T12:32:31.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.198+0000 7f7522ffd640 1 -- 172.21.15.3:0/602019054 >> v2:172.21.15.3:3300/0 conn(0x7f7524070650 msgr2=0x7f7524149f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:31.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.198+0000 7f7522ffd640 1 --2- 172.21.15.3:0/602019054 >> v2:172.21.15.3:3300/0 conn(0x7f7524070650 0x7f7524149f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:31.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.198+0000 7f7522ffd640 1 -- 172.21.15.3:0/602019054 >> v1:172.21.15.3:6789/0 conn(0x7f75240ac210 legacy=0x7f752406ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:31.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.198+0000 7f7522ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:32:31.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.198+0000 7f7522ffd640 1 monclient: found mon.noname-c 2023-10-30T12:32:31.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.198+0000 7f7522ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:31.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.198+0000 7f7522ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:32:31.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.198+0000 7f7522ffd640 1 -- 172.21.15.3:0/602019054 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75240ac660 con 0x7f7524142e40 2023-10-30T12:32:31.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.200+0000 7f7522ffd640 1 --2- 172.21.15.3:0/602019054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7524142e40 0x7f75240abcd0 secure :-1 s=READY pgs=5299 cs=0 l=1 rev1=1 crypto rx=0x7f752406bf70 tx=0x7f751c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79168f528bde69cc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:31.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f7521ffb640 1 -- 172.21.15.3:0/602019054 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751c031830 con 0x7f7524142e40 2023-10-30T12:32:31.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f7521ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:31.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f7521ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:32:31.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f7521ffb640 10 monclient: dump: 2023-10-30T12:32:31.212 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:31.212 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:31.212 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:31.214 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f7521ffb640 10 monclient: _finish_auth 0 2023-10-30T12:32:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f7521ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:32:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f7521ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f7521ffb640 1 -- 172.21.15.3:0/602019054 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751c02d040 con 0x7f7524142e40 2023-10-30T12:32:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f7521ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f7521ffb640 1 -- 172.21.15.3:0/602019054 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751c031bc0 con 0x7f7524142e40 2023-10-30T12:32:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f75298b4640 5 monclient: authenticate success, global_id 21799 2023-10-30T12:32:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f75298b4640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f75298b4640 4 set_mon_vals no callback set 2023-10-30T12:32:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f7521ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f7521ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f7521ffb640 10 monclient: dump: 2023-10-30T12:32:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:31.218 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f75298b4640 10 monclient: shutdown 2023-10-30T12:32:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f75298b4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f75298b4640 1 -- 172.21.15.3:0/602019054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7524142e40 msgr2=0x7f75240abcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f75298b4640 1 --2- 172.21.15.3:0/602019054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7524142e40 0x7f75240abcd0 secure :-1 s=READY pgs=5299 cs=0 l=1 rev1=1 crypto rx=0x7f752406bf70 tx=0x7f751c02b8d0 comp rx=0 tx=0).stop 2023-10-30T12:32:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f75298b4640 1 -- 172.21.15.3:0/602019054 shutdown_connections 2023-10-30T12:32:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f75298b4640 1 --2- 172.21.15.3:0/602019054 >> v2:172.21.15.3:3300/0 conn(0x7f7524070650 0x7f7524149f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f75298b4640 1 --2- 172.21.15.3:0/602019054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7524142e40 0x7f75240abcd0 unknown :-1 s=CLOSED pgs=5299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f75298b4640 1 -- 172.21.15.3:0/602019054 >> 172.21.15.3:0/602019054 conn(0x7f75240afb80 msgr2=0x7f75240aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f75298b4640 1 -- 172.21.15.3:0/602019054 shutdown_connections 2023-10-30T12:32:31.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.204+0000 7f75298b4640 1 -- 172.21.15.3:0/602019054 wait complete. 2023-10-30T12:32:31.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.205+0000 7f75298b4640 10 monclient: build_initial_monmap 2023-10-30T12:32:31.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.205+0000 7f75298b4640 10 monclient: monmap: 2023-10-30T12:32:31.220 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:31.220 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:31.221 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:31.221 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:31.221 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:31.221 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:31.221 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:31.221 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:31.222 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:31.222 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:31.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.205+0000 7f75298b4640 1 librados: starting msgr at 2023-10-30T12:32:31.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.205+0000 7f75298b4640 1 librados: starting objecter 2023-10-30T12:32:31.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.205+0000 7f75298b4640 1 Processor -- start 2023-10-30T12:32:31.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 1 -- start start 2023-10-30T12:32:31.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 1 librados: setting wanted keys 2023-10-30T12:32:31.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 1 librados: calling monclient init 2023-10-30T12:32:31.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 10 monclient: init 2023-10-30T12:32:31.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:31.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:32:31.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7524070650 0x7f752413c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:31.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 10 monclient(hunting): picked mon.noname-a con 0x7f7524070650 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:31.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f75240ac210 0x7f75240ab360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:31.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 10 monclient(hunting): picked mon.noname-c con 0x7f75240ac210 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:31.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7524142e40 0x7f75241d8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:31.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 10 monclient(hunting): picked mon.noname-b con 0x7f7524142e40 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:31.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:31.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75240acba0 con 0x7f7524070650 2023-10-30T12:32:31.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:31.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75241d81d0 con 0x7f7524142e40 2023-10-30T12:32:31.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75241d7ea0 con 0x7f75240ac210 2023-10-30T12:32:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.206+0000 7f75298b4640 10 monclient(hunting): authenticate will time out at 2742.420833s 2023-10-30T12:32:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f7522ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7524070650 0x7f752413c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f75227fc640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f75240ac210 0x7f75240ab360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f75237fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7524142e40 0x7f75241d8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f7522ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7524070650 0x7f752413c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56654/0 (socket says 172.21.15.3:56654) 2023-10-30T12:32:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f7522ffd640 1 -- 172.21.15.3:0/1571071498 learned_addr learned my addr 172.21.15.3:0/1571071498 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f7522ffd640 10 monclient(hunting): get_auth_request con 0x7f7524070650 auth_method 0 2023-10-30T12:32:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f7522ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:31.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f7522ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:31.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f7522ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:31.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f75227fc640 10 monclient(hunting): get_auth_request con 0x7f75240ac210 auth_method 0 2023-10-30T12:32:31.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f75227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:31.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f75227fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:31.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f75227fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:31.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f75237fe640 10 monclient(hunting): get_auth_request con 0x7f7524142e40 auth_method 0 2023-10-30T12:32:31.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f75237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:31.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f75237fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:31.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.207+0000 7f75237fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:31.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.208+0000 7f75237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:31.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.208+0000 7f75237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:31.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.208+0000 7f75237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:31.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.208+0000 7f7522ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:31.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.208+0000 7f7522ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:31.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.208+0000 7f7522ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:31.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.208+0000 7f75227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:31.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.208+0000 7f75227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:31.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.208+0000 7f75227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:31.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.210+0000 7f75237fe640 10 monclient(hunting): handle_auth_done global_id 21802 payload 824 2023-10-30T12:32:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.210+0000 7f75237fe640 1 -- 172.21.15.3:0/1571071498 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f75240ac210 msgr2=0x7f75240ab360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.210+0000 7f75237fe640 1 --2- 172.21.15.3:0/1571071498 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f75240ac210 0x7f75240ab360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.210+0000 7f75237fe640 1 -- 172.21.15.3:0/1571071498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7524070650 msgr2=0x7f752413c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.210+0000 7f75237fe640 1 --2- 172.21.15.3:0/1571071498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7524070650 0x7f752413c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.210+0000 7f75237fe640 10 monclient: _finish_hunting 0 2023-10-30T12:32:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.210+0000 7f75237fe640 1 monclient: found mon.noname-b 2023-10-30T12:32:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.210+0000 7f75237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.210+0000 7f75237fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.210+0000 7f75237fe640 1 -- 172.21.15.3:0/1571071498 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75240aa720 con 0x7f7524142e40 2023-10-30T12:32:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.212+0000 7f75237fe640 1 --2- 172.21.15.3:0/1571071498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7524142e40 0x7f75241d8660 secure :-1 s=READY pgs=5300 cs=0 l=1 rev1=1 crypto rx=0x7f751400efc0 tx=0x7f751400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.212+0000 7f750bfff640 10 client.?.objecter ms_handle_connect 0x7f7524142e40 2023-10-30T12:32:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.212+0000 7f750bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f750bfff640 1 -- 172.21.15.3:0/1571071498 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751400f040 con 0x7f7524142e40 2023-10-30T12:32:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f750bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f750bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f750bfff640 10 monclient: dump: 2023-10-30T12:32:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:31.237 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f750bfff640 10 monclient: _finish_auth 0 2023-10-30T12:32:31.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f750bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:32:31.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f750bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:31.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f750bfff640 1 -- 172.21.15.3:0/1571071498 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7514014480 con 0x7f7524142e40 2023-10-30T12:32:31.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f750bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:31.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f750bfff640 1 -- 172.21.15.3:0/1571071498 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7514014840 con 0x7f7524142e40 2023-10-30T12:32:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f750bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f750bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f750bfff640 10 monclient: dump: 2023-10-30T12:32:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:31.240 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:31.240 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:31.240 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:31.240 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:31.240 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:31.240 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:31.241 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:31.241 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:31.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f75298b4640 5 monclient: authenticate success, global_id 21802 2023-10-30T12:32:31.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f75298b4640 10 monclient: _renew_subs 2023-10-30T12:32:31.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f75298b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f75298b4640 1 -- 172.21.15.3:0/1571071498 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75241d8ba0 con 0x7f7524142e40 2023-10-30T12:32:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f75217fa640 4 set_mon_vals no callback set 2023-10-30T12:32:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f75298b4640 10 client.21802.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f75298b4640 10 monclient: _renew_subs 2023-10-30T12:32:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f75298b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f75298b4640 1 -- 172.21.15.3:0/1571071498 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7524139df0 con 0x7f7524142e40 2023-10-30T12:32:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.216+0000 7f75298b4640 1 librados: init done 2023-10-30T12:32:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.217+0000 7f75298b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.217+0000 7f75298b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.217+0000 7f75298b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:31.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.217+0000 7f75298b4640 1 -- 172.21.15.3:0/1571071498 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75241d8320 con 0x7f7524142e40 2023-10-30T12:32:31.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.235+0000 7f750bfff640 1 -- 172.21.15.3:0/1571071498 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f75140149b0 con 0x7f7524142e40 2023-10-30T12:32:31.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.236+0000 7f750bfff640 1 --2- 172.21.15.3:0/1571071498 >> v2:172.21.15.92:6800/1750155735 conn(0x7f751006c7f0 0x7f751006ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:31.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.236+0000 7f7522ffd640 1 --2- 172.21.15.3:0/1571071498 >> v2:172.21.15.92:6800/1750155735 conn(0x7f751006c7f0 0x7f751006ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:31.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.236+0000 7f7522ffd640 10 monclient: get_auth_request con 0x7f751006c7f0 auth_method 0 2023-10-30T12:32:31.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.236+0000 7f7522ffd640 1 --2- 172.21.15.3:0/1571071498 >> v2:172.21.15.92:6800/1750155735 conn(0x7f751006c7f0 0x7f751006ecb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f751c009560 tx=0x7f751c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:31.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.236+0000 7f750bfff640 10 client.21802.objecter ms_handle_connect 0x7f751006c7f0 2023-10-30T12:32:31.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.457+0000 7f750bfff640 1 -- 172.21.15.3:0/1571071498 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(899..899 src has 627..899) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f7514059300 con 0x7f7524142e40 2023-10-30T12:32:31.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.457+0000 7f750bfff640 10 client.21802.objecter ms_dispatch 0x7f7524000b90 osd_map(899..899 src has 627..899) v4 2023-10-30T12:32:31.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.457+0000 7f750bfff640 3 client.21802.objecter handle_osd_map got epochs [899,899] > 0 2023-10-30T12:32:31.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.457+0000 7f750bfff640 3 client.21802.objecter handle_osd_map decoding full epoch 899 2023-10-30T12:32:31.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.457+0000 7f750bfff640 20 client.21802.objecter dump_active .. 0 homeless 2023-10-30T12:32:31.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.466+0000 7f750bfff640 1 -- 172.21.15.3:0/1571071498 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f751405d3a0 con 0x7f7524142e40 2023-10-30T12:32:31.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.466+0000 7f750bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:31.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.466+0000 7f750bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:32:31.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.645+0000 7f75298b4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:32:31.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.645+0000 7f75298b4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:32:31.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.645+0000 7f75298b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:31.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.645+0000 7f75298b4640 1 -- 172.21.15.3:0/1571071498 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75241d8ba0 con 0x7f7524142e40 2023-10-30T12:32:31.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.676+0000 7f750bfff640 1 -- 172.21.15.3:0/1571071498 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v899) v1 ==== 74+0+15410 (secure 0 0 0) 0x7f7514061550 con 0x7f7524142e40 2023-10-30T12:32:31.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.676+0000 7f750bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:32:31.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.676+0000 7f750bfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:32:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:32:31.679 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":899,"fsid":"9bae44ae-7379-4ab2-a454-4d550607110c","created":"2023-10-30T12:09:17.680340+0000","modified":"2023-10-30T12:32:28.341670+0000","last_up_change":"2023-10-30T12:10:08.115093+0000","last_in_change":"2023-10-30T12:09:24.800567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":97,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:10:06.065827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T12:10:14.411118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"ceph_test_rados_api_asio","create_time":"2023-10-30T12:11:42.636830+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"69005df2-28b6-4bb2-b3e6-79df0d403c70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":892,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3639283465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6809","nonce":3639283465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6811","nonce":3639283465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3639283465}]},"public_addr":"172.21.15.3:6808/3639283465","cluster_addr":"172.21.15.3:6809/3639283465","heartbeat_back_addr":"172.21.15.3:6811/3639283465","heartbeat_front_addr":"172.21.15.3:6810/3639283465","state":["exists","up"]},{"osd":1,"uuid":"6466ea5c-d74a-4709-a83a-f5d4fe17157e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":892,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":1445650280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6805","nonce":1445650280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6807","nonce":1445650280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":1445650280}]},"public_addr":"172.21.15.3:6804/1445650280","cluster_addr":"172.21.15.3:6805/1445650280","heartbeat_back_addr":"172.21.15.3:6807/1445650280","heartbeat_front_addr":"172.21.15.3:6806/1445650280","state":["exists","up"]},{"osd":2,"uuid":"49f99cdc-c9e9-4b90-a66c-2f4fea52a3b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":892,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":1407765917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6801","nonce":1407765917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6803","nonce":1407765917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":1407765917}]},"public_addr":"172.21.15.3:6800/1407765917","cluster_addr":"172.21.15.3:6801/1407765917","heartbeat_back_addr":"172.21.15.3:6803/1407765917","heartbeat_front_addr":"172.21.15.3:6802/1407765917","state":["exists","up"]},{"osd":3,"uuid":"69ec5657-9972-4f4b-8012-83886eb47429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":892,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2654726311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":2654726311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6815","nonce":2654726311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2654726311}]},"public_addr":"172.21.15.3:6812/2654726311","cluster_addr":"172.21.15.3:6813/2654726311","heartbeat_back_addr":"172.21.15.3:6815/2654726311","heartbeat_front_addr":"172.21.15.3:6814/2654726311","state":["exists","up"]},{"osd":4,"uuid":"da3dad24-4aeb-47a3-ad1d-d13c77cafad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":888,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6801","nonce":4229938171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4229938171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4229938171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6803","nonce":4229938171}]},"public_addr":"172.21.15.92:6801/4229938171","cluster_addr":"172.21.15.92:6802/4229938171","heartbeat_back_addr":"172.21.15.92:6804/4229938171","heartbeat_front_addr":"172.21.15.92:6803/4229938171","state":["exists","up"]},{"osd":5,"uuid":"d38d9d7f-b0de-48ab-a850-499c6a0a4364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":892,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6805","nonce":3208162957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3208162957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3208162957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6807","nonce":3208162957}]},"public_addr":"172.21.15.92:6805/3208162957","cluster_addr":"172.21.15.92:6806/3208162957","heartbeat_back_addr":"172.21.15.92:6808/3208162957","heartbeat_front_addr":"172.21.15.92:6807/3208162957","state":["exists","up"]},{"osd":6,"uuid":"b39d2b0a-84ef-4909-bac7-47f146305a79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":892,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":3275842310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":3275842310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":3275842310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6811","nonce":3275842310}]},"public_addr":"172.21.15.92:6809/3275842310","cluster_addr":"172.21.15.92:6810/3275842310","heartbeat_back_addr":"172.21.15.92:6812/3275842310","heartbeat_front_addr":"172.21.15.92:6811/3275842310","state":["exists","up"]},{"osd":7,"uuid":"0a9d81ac-5a14-4fef-877a-982594787dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":892,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":1101825516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1101825516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1101825516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6815","nonce":1101825516}]},"public_addr":"172.21.15.92:6813/1101825516","cluster_addr":"172.21.15.92:6814/1101825516","heartbeat_back_addr":"172.21.15.92:6816/1101825516","heartbeat_front_addr":"172.21.15.92:6815/1101825516","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.340529+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.109770+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.885514+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:04.651007+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:02.744160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:03.156206+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:03.018599+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:10:03.334416+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":15,"snaps":[{"begin":9,"length":2}]},{"pool":52,"snaps":[{"begin":6,"length":2},{"begin":12,"length":2}]},{"pool":84,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":94,"snaps":[{"begin":6,"length":2}]},{"pool":105,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":117,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":175,"snaps":[{"begin":12,"length":1}]},{"pool":184,"snaps":[{"begin":14,"length":1}]},{"pool":190,"snaps":[{"begin":17,"length":1}]},{"pool":192,"snaps":[{"begin":19,"length":2}]},{"pool":197,"snaps":[{"begin":24,"length":1}]},{"pool":240,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":4,"length":1}]},{"pool":244,"snaps":[{"begin":6,"length":1}]},{"pool":248,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:32:31.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 10 librados: watch_flush enter 2023-10-30T12:32:31.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 10 librados: watch_flush exit 2023-10-30T12:32:31.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 20 client.21802.objecter shutdown clearing up homeless session... 2023-10-30T12:32:31.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 10 client.21802.objecter successfully canceled tick 2023-10-30T12:32:31.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 1 -- 172.21.15.3:0/1571071498 >> v2:172.21.15.92:6800/1750155735 conn(0x7f751006c7f0 msgr2=0x7f751006ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:31.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 1 --2- 172.21.15.3:0/1571071498 >> v2:172.21.15.92:6800/1750155735 conn(0x7f751006c7f0 0x7f751006ecb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f751c009560 tx=0x7f751c03c040 comp rx=0 tx=0).stop 2023-10-30T12:32:31.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 10 monclient: shutdown 2023-10-30T12:32:31.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:31.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 1 -- 172.21.15.3:0/1571071498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7524142e40 msgr2=0x7f75241d8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:31.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 1 --2- 172.21.15.3:0/1571071498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7524142e40 0x7f75241d8660 secure :-1 s=READY pgs=5300 cs=0 l=1 rev1=1 crypto rx=0x7f751400efc0 tx=0x7f751400c490 comp rx=0 tx=0).stop 2023-10-30T12:32:31.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 1 -- 172.21.15.3:0/1571071498 shutdown_connections 2023-10-30T12:32:31.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 1 --2- 172.21.15.3:0/1571071498 >> v2:172.21.15.92:6800/1750155735 conn(0x7f751006c7f0 0x7f751006ecb0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:31.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 1 --2- 172.21.15.3:0/1571071498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7524142e40 0x7f75241d8660 unknown :-1 s=CLOSED pgs=5300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:31.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 1 --2- 172.21.15.3:0/1571071498 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f75240ac210 0x7f75240ab360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:31.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 1 --2- 172.21.15.3:0/1571071498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7524070650 0x7f752413c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:31.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 1 -- 172.21.15.3:0/1571071498 >> 172.21.15.3:0/1571071498 conn(0x7f75240afb80 msgr2=0x7f7524142470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:31.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 1 -- 172.21.15.3:0/1571071498 shutdown_connections 2023-10-30T12:32:31.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 1 -- 172.21.15.3:0/1571071498 wait complete. 2023-10-30T12:32:31.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:31.678+0000 7f75298b4640 1 librados: shutdown 2023-10-30T12:32:32.692 INFO:tasks.ceph:Scrubbing osd.0 2023-10-30T12:32:32.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:32:32.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.809+0000 7f91795c5640 10 monclient: get_monmap_and_config 2023-10-30T12:32:32.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.809+0000 7f91795c5640 10 monclient: build_initial_monmap 2023-10-30T12:32:32.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.809+0000 7f91795c5640 10 monclient: monmap: 2023-10-30T12:32:32.811 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:32.812 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:32.812 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:32.810135+0000 2023-10-30T12:32:32.812 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:32.810135+0000 2023-10-30T12:32:32.812 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:32.812 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:32.813 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:32.813 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:32.813 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:32.813 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:32.813 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:32.813 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:32.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.809+0000 7f91795c5640 1 Processor -- start 2023-10-30T12:32:32.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.809+0000 7f91795c5640 1 -- start start 2023-10-30T12:32:32.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.809+0000 7f91795c5640 10 monclient: init 2023-10-30T12:32:32.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.810+0000 7f91795c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:32.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.810+0000 7f91795c5640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2023-10-30T12:32:32.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.810+0000 7f91795c5640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f91740b4420 0x7f917413ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:32.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.810+0000 7f91795c5640 10 monclient(hunting): picked mon.noname-c con 0x7f91740b4420 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:32.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.810+0000 7f91795c5640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f91740b12d0 0x7f917413ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:32.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.810+0000 7f91795c5640 10 monclient(hunting): picked mon.noname-a con 0x7f91740b12d0 addr v2:172.21.15.3:3300/0 2023-10-30T12:32:32.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.810+0000 7f91795c5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9174141910 addr v1:172.21.15.3:6789/0 2023-10-30T12:32:32.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.810+0000 7f91795c5640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91740a9120 con 0x7f9174141910 2023-10-30T12:32:32.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.810+0000 7f91795c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:32.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.810+0000 7f91795c5640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9174067a10 con 0x7f91740b12d0 2023-10-30T12:32:32.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.810+0000 7f91795c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:32.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.810+0000 7f91795c5640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91740b4800 con 0x7f91740b4420 2023-10-30T12:32:32.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.810+0000 7f91795c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:32.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.810+0000 7f91795c5640 10 monclient(hunting): authenticate will time out at 2744.024551s 2023-10-30T12:32:32.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.811+0000 7f91727fc640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f91740b12d0 0x7f917413ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:32.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.811+0000 7f9172ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f91740b4420 0x7f917413ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:32.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.811+0000 7f91737fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9174141910 0x7f9174141d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:48030/0 (socket says 172.21.15.3:48030) 2023-10-30T12:32:32.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.811+0000 7f91737fe640 1 -- 172.21.15.3:0/1441385653 learned_addr learned my addr 172.21.15.3:0/1441385653 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:32:32.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.811+0000 7f91727fc640 10 monclient(hunting): get_auth_request con 0x7f91740b12d0 auth_method 0 2023-10-30T12:32:32.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.811+0000 7f91727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:32.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.811+0000 7f91727fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:32.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.811+0000 7f91727fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:32.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.811+0000 7f9172ffd640 10 monclient(hunting): get_auth_request con 0x7f91740b4420 auth_method 0 2023-10-30T12:32:32.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.811+0000 7f9172ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:32.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.811+0000 7f9172ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:32.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.811+0000 7f9172ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:32.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.812+0000 7f9172ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:32.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.812+0000 7f9172ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:32.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.812+0000 7f9172ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:32.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.812+0000 7f91727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:32.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.812+0000 7f91727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:32.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.812+0000 7f91727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:32.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.813+0000 7f9172ffd640 10 monclient(hunting): handle_auth_done global_id 21805 payload 274 2023-10-30T12:32:32.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.813+0000 7f9172ffd640 1 -- 172.21.15.3:0/1441385653 >> v2:172.21.15.3:3300/0 conn(0x7f91740b12d0 msgr2=0x7f917413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:32.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.813+0000 7f9172ffd640 1 --2- 172.21.15.3:0/1441385653 >> v2:172.21.15.3:3300/0 conn(0x7f91740b12d0 0x7f917413ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.813+0000 7f9172ffd640 1 -- 172.21.15.3:0/1441385653 >> v1:172.21.15.3:6789/0 conn(0x7f9174141910 legacy=0x7f9174141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:32.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.813+0000 7f9172ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:32:32.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.813+0000 7f9172ffd640 1 monclient: found mon.noname-c 2023-10-30T12:32:32.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.813+0000 7f9172ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:32.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.813+0000 7f9172ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:32:32.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.813+0000 7f9172ffd640 1 -- 172.21.15.3:0/1441385653 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91740aa460 con 0x7f91740b4420 2023-10-30T12:32:32.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.815+0000 7f9172ffd640 1 --2- 172.21.15.3:0/1441385653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91740b4420 0x7f917413ea50 secure :-1 s=READY pgs=5301 cs=0 l=1 rev1=1 crypto rx=0x7f91740b4180 tx=0x7f916c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e0d4d743e7bc420 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:32.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f9171ffb640 1 -- 172.21.15.3:0/1441385653 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916c0315a0 con 0x7f91740b4420 2023-10-30T12:32:32.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f9171ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:32.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f9171ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:32:32.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f9171ffb640 10 monclient: dump: 2023-10-30T12:32:32.831 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:32.832 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:32.832 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:32.832 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:32.832 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:32.832 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:32.833 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:32.833 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:32.833 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:32.833 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:32.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f9171ffb640 10 monclient: _finish_auth 0 2023-10-30T12:32:32.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f9171ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:32:32.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f9171ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:32.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f9171ffb640 1 -- 172.21.15.3:0/1441385653 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916c031710 con 0x7f91740b4420 2023-10-30T12:32:32.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f91795c5640 5 monclient: authenticate success, global_id 21805 2023-10-30T12:32:32.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f91795c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:32:32.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f9171ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:32.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f91795c5640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:32.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f9171ffb640 1 -- 172.21.15.3:0/1441385653 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916c031ab0 con 0x7f91740b4420 2023-10-30T12:32:32.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f9171ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:32.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f91795c5640 4 set_mon_vals no callback set 2023-10-30T12:32:32.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f9171ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:32.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f9171ffb640 10 monclient: dump: 2023-10-30T12:32:32.836 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:32.836 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:32.836 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:32.836 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:32.837 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:32.837 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:32.837 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:32.837 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:32.837 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:32.837 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:32.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f91795c5640 10 monclient: shutdown 2023-10-30T12:32:32.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f91795c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:32.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f91795c5640 1 -- 172.21.15.3:0/1441385653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91740b4420 msgr2=0x7f917413ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:32.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.819+0000 7f91795c5640 1 --2- 172.21.15.3:0/1441385653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91740b4420 0x7f917413ea50 secure :-1 s=READY pgs=5301 cs=0 l=1 rev1=1 crypto rx=0x7f91740b4180 tx=0x7f916c02b840 comp rx=0 tx=0).stop 2023-10-30T12:32:32.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.820+0000 7f91795c5640 1 -- 172.21.15.3:0/1441385653 shutdown_connections 2023-10-30T12:32:32.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.820+0000 7f91795c5640 1 --2- 172.21.15.3:0/1441385653 >> v2:172.21.15.3:3300/0 conn(0x7f91740b12d0 0x7f917413ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.820+0000 7f91795c5640 1 --2- 172.21.15.3:0/1441385653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91740b4420 0x7f917413ea50 unknown :-1 s=CLOSED pgs=5301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.820+0000 7f91795c5640 1 -- 172.21.15.3:0/1441385653 >> 172.21.15.3:0/1441385653 conn(0x7f9174065ea0 msgr2=0x7f91740662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:32.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.820+0000 7f91795c5640 1 -- 172.21.15.3:0/1441385653 shutdown_connections 2023-10-30T12:32:32.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.820+0000 7f91795c5640 1 -- 172.21.15.3:0/1441385653 wait complete. 2023-10-30T12:32:32.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.820+0000 7f91795c5640 10 monclient: build_initial_monmap 2023-10-30T12:32:32.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.820+0000 7f91795c5640 10 monclient: monmap: 2023-10-30T12:32:32.840 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:32.840 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:32.840 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:32.841 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:32.841 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:32.841 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:32.841 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:32.841 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:32.842 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:32.842 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:32.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.820+0000 7f91795c5640 1 librados: starting msgr at 2023-10-30T12:32:32.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.820+0000 7f91795c5640 1 librados: starting objecter 2023-10-30T12:32:32.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 1 Processor -- start 2023-10-30T12:32:32.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 1 -- start start 2023-10-30T12:32:32.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 1 librados: setting wanted keys 2023-10-30T12:32:32.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 1 librados: calling monclient init 2023-10-30T12:32:32.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 10 monclient: init 2023-10-30T12:32:32.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:32.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:32:32.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91740b12d0 0x7f91741392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:32.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 10 monclient(hunting): picked mon.noname-b con 0x7f91740b12d0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:32.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91740b4420 0x7f917413ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:32.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 10 monclient(hunting): picked mon.noname-c con 0x7f91740b4420 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:32.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9174141910 0x7f91741d6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:32.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 10 monclient(hunting): picked mon.noname-a con 0x7f9174141910 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:32.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:32.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9174066510 con 0x7f9174141910 2023-10-30T12:32:32.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:32.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91741d67b0 con 0x7f91740b12d0 2023-10-30T12:32:32.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:32.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91741d6a90 con 0x7f91740b4420 2023-10-30T12:32:32.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:32.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.821+0000 7f91795c5640 10 monclient(hunting): authenticate will time out at 2744.035997s 2023-10-30T12:32:32.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f91737fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9174141910 0x7f91741d6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:32.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f9172ffd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91740b12d0 0x7f91741392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:32.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f91727fc640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91740b4420 0x7f917413ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:32.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f91737fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9174141910 0x7f91741d6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40386/0 (socket says 172.21.15.3:40386) 2023-10-30T12:32:32.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f91737fe640 1 -- 172.21.15.3:0/496396694 learned_addr learned my addr 172.21.15.3:0/496396694 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:32.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f91737fe640 10 monclient(hunting): get_auth_request con 0x7f9174141910 auth_method 0 2023-10-30T12:32:32.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f91737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:32.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f91737fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:32.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f91737fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:32.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f9172ffd640 10 monclient(hunting): get_auth_request con 0x7f91740b12d0 auth_method 0 2023-10-30T12:32:32.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f9172ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:32.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f9172ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:32.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f9172ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:32.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f91727fc640 10 monclient(hunting): get_auth_request con 0x7f91740b4420 auth_method 0 2023-10-30T12:32:32.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f91727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:32.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f91727fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:32.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.822+0000 7f91727fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:32.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.823+0000 7f91737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:32.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.823+0000 7f91737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:32.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.823+0000 7f91737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:32.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.823+0000 7f9172ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:32.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.823+0000 7f9172ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:32.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.823+0000 7f9172ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:32.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.823+0000 7f91727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:32.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.823+0000 7f91727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:32.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.823+0000 7f91727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:32.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.825+0000 7f9172ffd640 10 monclient(hunting): handle_auth_done global_id 21808 payload 824 2023-10-30T12:32:32.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.825+0000 7f9172ffd640 1 -- 172.21.15.3:0/496396694 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91740b4420 msgr2=0x7f917413ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:32.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.825+0000 7f9172ffd640 1 --2- 172.21.15.3:0/496396694 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91740b4420 0x7f917413ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.825+0000 7f9172ffd640 1 -- 172.21.15.3:0/496396694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9174141910 msgr2=0x7f91741d6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:32.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.825+0000 7f9172ffd640 1 --2- 172.21.15.3:0/496396694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9174141910 0x7f91741d6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.825+0000 7f9172ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:32:32.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.825+0000 7f9172ffd640 1 monclient: found mon.noname-b 2023-10-30T12:32:32.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.825+0000 7f9172ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:32.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.825+0000 7f9172ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:32.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.825+0000 7f9172ffd640 1 -- 172.21.15.3:0/496396694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917413cbe0 con 0x7f91740b12d0 2023-10-30T12:32:32.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.827+0000 7f9172ffd640 1 --2- 172.21.15.3:0/496396694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91740b12d0 0x7f91741392d0 secure :-1 s=READY pgs=5302 cs=0 l=1 rev1=1 crypto rx=0x7f916c033000 tx=0x7f916c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:32.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.827+0000 7f916b7fe640 10 client.?.objecter ms_handle_connect 0x7f91740b12d0 2023-10-30T12:32:32.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.827+0000 7f916b7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:32.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f916b7fe640 1 -- 172.21.15.3:0/496396694 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916c032840 con 0x7f91740b12d0 2023-10-30T12:32:32.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f916b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:32.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f916b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:32.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f916b7fe640 10 monclient: dump: 2023-10-30T12:32:32.857 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:32.857 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:32.857 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:32.858 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:32.858 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:32.858 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:32.858 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:32.858 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:32.859 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:32.859 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:32.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f916b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:32:32.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f916b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:32:32.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f916b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:32.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f916b7fe640 1 -- 172.21.15.3:0/496396694 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916c03f030 con 0x7f91740b12d0 2023-10-30T12:32:32.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f91795c5640 5 monclient: authenticate success, global_id 21808 2023-10-30T12:32:32.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f916b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:32.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f91795c5640 10 monclient: _renew_subs 2023-10-30T12:32:32.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f916b7fe640 1 -- 172.21.15.3:0/496396694 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916c032bd0 con 0x7f91740b12d0 2023-10-30T12:32:32.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f91795c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:32.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f91795c5640 1 -- 172.21.15.3:0/496396694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f917413e350 con 0x7f91740b12d0 2023-10-30T12:32:32.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f916b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:32.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f9170ff9640 4 set_mon_vals no callback set 2023-10-30T12:32:32.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f916b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:32.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f916b7fe640 10 monclient: dump: 2023-10-30T12:32:32.862 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:32.862 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:32.862 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:32.862 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:32.863 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:32.863 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:32.863 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:32.863 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:32.863 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:32.864 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:32.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f91795c5640 10 client.21808.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:32.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f91795c5640 10 monclient: _renew_subs 2023-10-30T12:32:32.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f91795c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:32.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f91795c5640 1 -- 172.21.15.3:0/496396694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f917413e130 con 0x7f91740b12d0 2023-10-30T12:32:32.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f91795c5640 1 librados: init done 2023-10-30T12:32:32.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f91795c5640 10 client.21808.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:32.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f91795c5640 15 client.21808.objecter _session_command_op_assign -1 1 2023-10-30T12:32:32.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f91795c5640 20 client.21808.objecter _get_session osd=-1 returning homeless 2023-10-30T12:32:32.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f91795c5640 10 client.21808.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:32.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f91795c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:32.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.831+0000 7f91795c5640 1 -- 172.21.15.3:0/496396694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f91740aa740 con 0x7f91740b12d0 2023-10-30T12:32:32.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.850+0000 7f916b7fe640 1 -- 172.21.15.3:0/496396694 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f916c032d40 con 0x7f91740b12d0 2023-10-30T12:32:32.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.850+0000 7f916b7fe640 1 --2- 172.21.15.3:0/496396694 >> v2:172.21.15.92:6800/1750155735 conn(0x7f915406c810 0x7f915406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:32.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.850+0000 7f91727fc640 1 --2- 172.21.15.3:0/496396694 >> v2:172.21.15.92:6800/1750155735 conn(0x7f915406c810 0x7f915406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:32.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.850+0000 7f91727fc640 10 monclient: get_auth_request con 0x7f915406c810 auth_method 0 2023-10-30T12:32:32.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.851+0000 7f91727fc640 1 --2- 172.21.15.3:0/496396694 >> v2:172.21.15.92:6800/1750155735 conn(0x7f915406c810 0x7f915406ecd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f917413a460 tx=0x7f915c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:32.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.851+0000 7f916b7fe640 10 client.21808.objecter ms_handle_connect 0x7f915406c810 2023-10-30T12:32:32.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 1 -- 172.21.15.3:0/496396694 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(899..899 src has 627..899) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f916c07a060 con 0x7f91740b12d0 2023-10-30T12:32:32.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 10 client.21808.objecter ms_dispatch 0x7f9174000b90 osd_map(899..899 src has 627..899) v4 2023-10-30T12:32:32.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 3 client.21808.objecter handle_osd_map got epochs [899,899] > 0 2023-10-30T12:32:32.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 3 client.21808.objecter handle_osd_map decoding full epoch 899 2023-10-30T12:32:32.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 10 client.21808.objecter checking command 1 2023-10-30T12:32:32.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 1 --2- 172.21.15.3:0/496396694 >> v2:172.21.15.3:6808/3639283465 conn(0x7f9154073a80 0x7f9154075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:32.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 20 client.21808.objecter _get_session s=0x7f916c032bd0 osd=0 3 2023-10-30T12:32:32.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 20 client.21808.objecter put_session s=0x7f916c032bd0 osd=0 3 2023-10-30T12:32:32.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 15 client.21808.objecter _session_command_op_remove -1 1 2023-10-30T12:32:32.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 20 client.21808.objecter _get_session s=0x7f916c032bd0 osd=0 3 2023-10-30T12:32:32.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 20 client.21808.objecter get_session s=0x7f916c032bd0 osd=0 3 2023-10-30T12:32:32.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 15 client.21808.objecter _session_command_op_assign 0 1 2023-10-30T12:32:32.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 20 client.21808.objecter put_session s=0x7f916c032bd0 osd=0 4 2023-10-30T12:32:32.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 10 client.21808.objecter _send_command 1 2023-10-30T12:32:32.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 1 -- 172.21.15.3:0/496396694 --> v2:172.21.15.3:6808/3639283465 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f916c032840 con 0x7f9154073a80 2023-10-30T12:32:32.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 20 client.21808.objecter dump_active .. 0 homeless 2023-10-30T12:32:32.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 1 -- 172.21.15.3:0/496396694 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=899) v2 ==== 24+0+0 (secure 0 0 0) 0x7f916c02e070 con 0x7f91740b12d0 2023-10-30T12:32:32.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f916b7fe640 10 monclient: handle_get_version_reply finishing 1 version 899 2023-10-30T12:32:32.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.852+0000 7f91737fe640 1 --2- 172.21.15.3:0/496396694 >> v2:172.21.15.3:6808/3639283465 conn(0x7f9154073a80 0x7f9154075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:32.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.853+0000 7f91737fe640 10 monclient: get_auth_request con 0x7f9154073a80 auth_method 0 2023-10-30T12:32:32.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.855+0000 7f91737fe640 1 --2- 172.21.15.3:0/496396694 >> v2:172.21.15.3:6808/3639283465 conn(0x7f9154073a80 0x7f9154075f20 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:32.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.855+0000 7f916b7fe640 10 client.21808.objecter ms_handle_connect 0x7f9154073a80 2023-10-30T12:32:32.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.880+0000 7f916b7fe640 1 -- 172.21.15.3:0/496396694 <== osd.0 v2:172.21.15.3:6808/3639283465 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f916c032840 con 0x7f9154073a80 2023-10-30T12:32:32.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.880+0000 7f916b7fe640 10 client.21808.objecter ms_dispatch 0x7f9174000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:32:32.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.880+0000 7f916b7fe640 10 client.21808.objecter _finish_command 1 = system:0 2023-10-30T12:32:32.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.880+0000 7f916b7fe640 20 client.21808.objecter put_session s=0x7f916c032bd0 osd=0 4 2023-10-30T12:32:32.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.880+0000 7f916b7fe640 15 client.21808.objecter _session_command_op_remove 0 1 2023-10-30T12:32:32.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.893+0000 7f91795c5640 10 client.21808.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:32:32.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.893+0000 7f91795c5640 15 client.21808.objecter _session_command_op_assign -1 2 2023-10-30T12:32:32.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.893+0000 7f91795c5640 20 client.21808.objecter _get_session s=0x7f916c032bd0 osd=0 3 2023-10-30T12:32:32.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.893+0000 7f91795c5640 20 client.21808.objecter put_session s=0x7f916c032bd0 osd=0 3 2023-10-30T12:32:32.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.893+0000 7f91795c5640 20 client.21808.objecter _get_session s=0x7f916c032bd0 osd=0 3 2023-10-30T12:32:32.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.893+0000 7f91795c5640 15 client.21808.objecter _session_command_op_remove -1 2 2023-10-30T12:32:32.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.893+0000 7f91795c5640 20 client.21808.objecter get_session s=0x7f916c032bd0 osd=0 3 2023-10-30T12:32:32.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.893+0000 7f91795c5640 15 client.21808.objecter _session_command_op_assign 0 2 2023-10-30T12:32:32.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.893+0000 7f91795c5640 20 client.21808.objecter put_session s=0x7f916c032bd0 osd=0 4 2023-10-30T12:32:32.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.893+0000 7f91795c5640 10 client.21808.objecter _send_command 2 2023-10-30T12:32:32.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.893+0000 7f91795c5640 1 -- 172.21.15.3:0/496396694 --> v2:172.21.15.3:6808/3639283465 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f91740aa460 con 0x7f9154073a80 2023-10-30T12:32:32.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f916b7fe640 1 -- 172.21.15.3:0/496396694 <== osd.0 v2:172.21.15.3:6808/3639283465 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f91740aa460 con 0x7f9154073a80 2023-10-30T12:32:32.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f916b7fe640 10 client.21808.objecter ms_dispatch 0x7f9174000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:32:32.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f916b7fe640 10 client.21808.objecter _finish_command 2 = system:0 2023-10-30T12:32:32.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f916b7fe640 20 client.21808.objecter put_session s=0x7f916c032bd0 osd=0 4 2023-10-30T12:32:32.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f916b7fe640 15 client.21808.objecter _session_command_op_remove 0 2 2023-10-30T12:32:32.910 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-10-30T12:32:32.910 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:32:32.910 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-10-30T12:32:32.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 10 librados: watch_flush enter 2023-10-30T12:32:32.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 10 librados: watch_flush exit 2023-10-30T12:32:32.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 10 client.21808.objecter close_session for osd.0 2023-10-30T12:32:32.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 1 -- 172.21.15.3:0/496396694 >> v2:172.21.15.3:6808/3639283465 conn(0x7f9154073a80 msgr2=0x7f9154075f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:32.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 1 --2- 172.21.15.3:0/496396694 >> v2:172.21.15.3:6808/3639283465 conn(0x7f9154073a80 0x7f9154075f20 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 20 client.21808.objecter put_session s=0x7f916c032bd0 osd=0 1 2023-10-30T12:32:32.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 20 client.21808.objecter shutdown clearing up homeless session... 2023-10-30T12:32:32.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 10 client.21808.objecter successfully canceled tick 2023-10-30T12:32:32.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 1 -- 172.21.15.3:0/496396694 >> v2:172.21.15.92:6800/1750155735 conn(0x7f915406c810 msgr2=0x7f915406ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:32.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 1 --2- 172.21.15.3:0/496396694 >> v2:172.21.15.92:6800/1750155735 conn(0x7f915406c810 0x7f915406ecd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f917413a460 tx=0x7f915c009190 comp rx=0 tx=0).stop 2023-10-30T12:32:32.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 10 monclient: shutdown 2023-10-30T12:32:32.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:32.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 1 -- 172.21.15.3:0/496396694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91740b12d0 msgr2=0x7f91741392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:32.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 1 --2- 172.21.15.3:0/496396694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91740b12d0 0x7f91741392d0 secure :-1 s=READY pgs=5302 cs=0 l=1 rev1=1 crypto rx=0x7f916c033000 tx=0x7f916c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91737fe640 1 -- 172.21.15.3:0/496396694 reap_dead start 2023-10-30T12:32:32.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 1 -- 172.21.15.3:0/496396694 shutdown_connections 2023-10-30T12:32:32.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 1 -- 172.21.15.3:0/496396694 >> 172.21.15.3:0/496396694 conn(0x7f9174065ea0 msgr2=0x7f91741cc840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:32.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.907+0000 7f91795c5640 1 -- 172.21.15.3:0/496396694 shutdown_connections 2023-10-30T12:32:32.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.908+0000 7f91795c5640 1 -- 172.21.15.3:0/496396694 wait complete. 2023-10-30T12:32:32.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:32.908+0000 7f91795c5640 1 librados: shutdown 2023-10-30T12:32:32.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-30T12:32:33.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.037+0000 7f1a368ec640 10 monclient: get_monmap_and_config 2023-10-30T12:32:33.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.037+0000 7f1a368ec640 10 monclient: build_initial_monmap 2023-10-30T12:32:33.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.037+0000 7f1a368ec640 10 monclient: monmap: 2023-10-30T12:32:33.039 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:33.040 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:33.040 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:33.038501+0000 2023-10-30T12:32:33.040 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:33.038501+0000 2023-10-30T12:32:33.040 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:33.040 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:33.040 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:33.041 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:33.041 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:33.041 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:33.041 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:33.041 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.038+0000 7f1a368ec640 1 Processor -- start 2023-10-30T12:32:33.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.038+0000 7f1a368ec640 1 -- start start 2023-10-30T12:32:33.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.038+0000 7f1a368ec640 10 monclient: init 2023-10-30T12:32:33.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.039+0000 7f1a368ec640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:33.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.039+0000 7f1a368ec640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2023-10-30T12:32:33.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.039+0000 7f1a368ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a300b4350 addr v1:172.21.15.92:6789/0 2023-10-30T12:32:33.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.039+0000 7f1a368ec640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1a3013eb50 0x7f1a3013efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.039+0000 7f1a368ec640 10 monclient(hunting): picked mon.noname-a con 0x7f1a3013eb50 addr v2:172.21.15.3:3300/0 2023-10-30T12:32:33.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.039+0000 7f1a368ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1a30144f50 addr v1:172.21.15.3:6789/0 2023-10-30T12:32:33.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.039+0000 7f1a368ec640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a300aa6e0 con 0x7f1a30144f50 2023-10-30T12:32:33.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.039+0000 7f1a368ec640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a300a8bc0 con 0x7f1a300b4350 2023-10-30T12:32:33.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.039+0000 7f1a368ec640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.039+0000 7f1a368ec640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a300679f0 con 0x7f1a3013eb50 2023-10-30T12:32:33.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.039+0000 7f1a368ec640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:33.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.039+0000 7f1a368ec640 10 monclient(hunting): authenticate will time out at 2744.253411s 2023-10-30T12:32:33.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.074+0000 7f1a2ffff640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f1a300b4350 0x7f1a300a83d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.3:47036/0 (socket says 172.21.15.3:47036) 2023-10-30T12:32:33.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.074+0000 7f1a2ffff640 1 -- 172.21.15.3:0/320334482 learned_addr learned my addr 172.21.15.3:0/320334482 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:32:33.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.079+0000 7f1a2effd640 1 -- 172.21.15.3:0/320334482 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860331564 0 0) 0x7f1a20001110 con 0x7f1a300b4350 2023-10-30T12:32:33.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.079+0000 7f1a2effd640 10 monclient(hunting): _negotiate 2023-10-30T12:32:33.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.079+0000 7f1a2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:33.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.079+0000 7f1a2effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:33.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.079+0000 7f1a2effd640 10 monclient(hunting): my global_id is 21811 2023-10-30T12:32:33.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.079+0000 7f1a2effd640 1 -- 172.21.15.3:0/320334482 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a18003520 con 0x7f1a300b4350 2023-10-30T12:32:33.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 1 -- 172.21.15.3:0/320334482 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3535742642 0 0) 0x7f1a20001510 con 0x7f1a300b4350 2023-10-30T12:32:33.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 1 -- 172.21.15.3:0/320334482 >> v2:172.21.15.3:3300/0 conn(0x7f1a3013eb50 msgr2=0x7f1a3013efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:33.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 1 --2- 172.21.15.3:0/320334482 >> v2:172.21.15.3:3300/0 conn(0x7f1a3013eb50 0x7f1a3013efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 1 -- 172.21.15.3:0/320334482 >> v1:172.21.15.3:6789/0 conn(0x7f1a30144f50 legacy=0x7f1a301453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:33.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient: _finish_hunting 0 2023-10-30T12:32:33.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:32:33.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:33.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.92:6789/0 2023-10-30T12:32:33.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 1 -- 172.21.15.3:0/320334482 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a300adf00 con 0x7f1a300b4350 2023-10-30T12:32:33.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient: _finish_auth 0 2023-10-30T12:32:33.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient: _check_auth_tickets 2023-10-30T12:32:33.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:33.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 1 -- 172.21.15.3:0/320334482 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1777801622 0 0) 0x7f1a200019b0 con 0x7f1a300b4350 2023-10-30T12:32:33.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:33.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:32:33.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient: dump: 2023-10-30T12:32:33.089 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:33.089 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:33.089 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.089 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.089 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:33.090 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:33.090 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:33.090 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:33.090 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:33.090 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] but i'm connected to v1:172.21.15.92:6789/0, reconnecting 2023-10-30T12:32:33.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:33.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 1 -- 172.21.15.3:0/320334482 >> v1:172.21.15.92:6789/0 conn(0x7f1a300b4350 legacy=0x7f1a300a83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:33.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:32:33.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 1 --2- 172.21.15.3:0/320334482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a180039c0 0x7f1a18003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient(hunting): picked mon.b con 0x7f1a180039c0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:33.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 1 --2- 172.21.15.3:0/320334482 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1a180047c0 0x7f1a18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient(hunting): picked mon.c con 0x7f1a180047c0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:33.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 1 --2- 172.21.15.3:0/320334482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a180053d0 0x7f1a1800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient(hunting): picked mon.a con 0x7f1a180053d0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:33.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 1 -- 172.21.15.3:0/320334482 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a300679f0 con 0x7f1a180053d0 2023-10-30T12:32:33.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 1 -- 172.21.15.3:0/320334482 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1800c310 con 0x7f1a180039c0 2023-10-30T12:32:33.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 1 -- 172.21.15.3:0/320334482 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1800c4e0 con 0x7f1a180047c0 2023-10-30T12:32:33.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:33.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.083+0000 7f1a2effd640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:32:33.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.084+0000 7f1a2f7fe640 1 --2- 172.21.15.3:0/320334482 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1a180047c0 0x7f1a18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.084+0000 7f1a2f7fe640 10 monclient(hunting): get_auth_request con 0x7f1a180047c0 auth_method 0 2023-10-30T12:32:33.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.084+0000 7f1a2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:33.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.084+0000 7f1a2f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:33.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.084+0000 7f1a2f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:32:33.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.085+0000 7f1a2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:33.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.085+0000 7f1a2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:33.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.085+0000 7f1a2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:32:33.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a2f7fe640 10 monclient(hunting): handle_auth_done global_id 21811 payload 293 2023-10-30T12:32:33.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a2f7fe640 1 -- 172.21.15.3:0/320334482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a180039c0 msgr2=0x7f1a18003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:33.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a2f7fe640 1 --2- 172.21.15.3:0/320334482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a180039c0 0x7f1a18003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a2f7fe640 1 -- 172.21.15.3:0/320334482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a180053d0 msgr2=0x7f1a1800ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:33.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a2f7fe640 1 --2- 172.21.15.3:0/320334482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a180053d0 0x7f1a1800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a2f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:32:33.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a2f7fe640 1 monclient: found mon.c 2023-10-30T12:32:33.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:33.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a2f7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:32:33.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a2f7fe640 1 -- 172.21.15.3:0/320334482 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a20001510 con 0x7f1a180047c0 2023-10-30T12:32:33.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a34e89640 1 -- 172.21.15.3:0/320334482 reap_dead start 2023-10-30T12:32:33.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a2f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:32:33.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a2f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:32:33.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:33.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a368ec640 5 monclient: authenticate success, global_id 21811 2023-10-30T12:32:33.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.088+0000 7f1a368ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:32:33.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.090+0000 7f1a2f7fe640 1 --2- 172.21.15.3:0/320334482 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1a180047c0 0x7f1a18004c40 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009860 tx=0x7f1a180057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b498ea7a744a936 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:33.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.155+0000 7f1a2effd640 1 -- 172.21.15.3:0/320334482 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2402e050 con 0x7f1a180047c0 2023-10-30T12:32:33.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.155+0000 7f1a2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:33.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.155+0000 7f1a2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:32:33.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.155+0000 7f1a2effd640 10 monclient: dump: 2023-10-30T12:32:33.157 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:33.158 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:33.158 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.158 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.158 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:33.158 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:33.159 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:33.159 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:33.159 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:33.159 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.155+0000 7f1a2effd640 1 -- 172.21.15.3:0/320334482 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a24032620 con 0x7f1a180047c0 2023-10-30T12:32:33.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.155+0000 7f1a368ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:32:33.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.155+0000 7f1a2effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:33.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.155+0000 7f1a368ec640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:33.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.155+0000 7f1a2effd640 1 -- 172.21.15.3:0/320334482 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a240329e0 con 0x7f1a180047c0 2023-10-30T12:32:33.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.155+0000 7f1a2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:33.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.155+0000 7f1a368ec640 4 set_mon_vals no callback set 2023-10-30T12:32:33.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.155+0000 7f1a2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:32:33.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.155+0000 7f1a2effd640 10 monclient: dump: 2023-10-30T12:32:33.161 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:33.161 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:33.162 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.162 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.162 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:33.162 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:33.162 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:33.163 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:33.163 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:33.163 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.156+0000 7f1a368ec640 10 monclient: shutdown 2023-10-30T12:32:33.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.156+0000 7f1a368ec640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:33.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.156+0000 7f1a368ec640 1 -- 172.21.15.3:0/320334482 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1a180047c0 msgr2=0x7f1a18004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:33.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.156+0000 7f1a368ec640 1 --2- 172.21.15.3:0/320334482 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1a180047c0 0x7f1a18004c40 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009860 tx=0x7f1a180057f0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.156+0000 7f1a368ec640 1 -- 172.21.15.3:0/320334482 shutdown_connections 2023-10-30T12:32:33.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.156+0000 7f1a368ec640 1 --2- 172.21.15.3:0/320334482 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1a180047c0 0x7f1a18004c40 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.156+0000 7f1a368ec640 1 -- 172.21.15.3:0/320334482 >> 172.21.15.3:0/320334482 conn(0x7f1a30065ea0 msgr2=0x7f1a300662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:33.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.156+0000 7f1a368ec640 1 -- 172.21.15.3:0/320334482 shutdown_connections 2023-10-30T12:32:33.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.156+0000 7f1a368ec640 1 -- 172.21.15.3:0/320334482 wait complete. 2023-10-30T12:32:33.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.157+0000 7f1a368ec640 10 monclient: build_initial_monmap 2023-10-30T12:32:33.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.157+0000 7f1a368ec640 10 monclient: monmap: 2023-10-30T12:32:33.169 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:33.169 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:33.169 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:33.169 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:33.169 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:33.170 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:33.170 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:33.170 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:33.170 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:33.170 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.157+0000 7f1a368ec640 1 librados: starting msgr at 2023-10-30T12:32:33.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.157+0000 7f1a368ec640 1 librados: starting objecter 2023-10-30T12:32:33.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.157+0000 7f1a368ec640 1 Processor -- start 2023-10-30T12:32:33.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.157+0000 7f1a368ec640 1 -- start start 2023-10-30T12:32:33.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.157+0000 7f1a368ec640 1 librados: setting wanted keys 2023-10-30T12:32:33.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.157+0000 7f1a368ec640 1 librados: calling monclient init 2023-10-30T12:32:33.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.157+0000 7f1a368ec640 10 monclient: init 2023-10-30T12:32:33.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:33.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:32:33.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1a180047c0 0x7f1a300af480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 10 monclient(hunting): picked mon.noname-c con 0x7f1a180047c0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:33.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a300b1160 0x7f1a30138400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 10 monclient(hunting): picked mon.noname-a con 0x7f1a300b1160 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:33.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a300af9c0 0x7f1a30138940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 10 monclient(hunting): picked mon.noname-b con 0x7f1a300af9c0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:33.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a300b2ef0 con 0x7f1a300b1160 2023-10-30T12:32:33.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a300aed10 con 0x7f1a300af9c0 2023-10-30T12:32:33.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a301d6e10 con 0x7f1a180047c0 2023-10-30T12:32:33.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:33.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.158+0000 7f1a368ec640 10 monclient(hunting): authenticate will time out at 2744.372337s 2023-10-30T12:32:33.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.159+0000 7f1a2f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a300b1160 0x7f1a30138400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.159+0000 7f1a34e89640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a300af9c0 0x7f1a30138940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.159+0000 7f1a2f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a300b1160 0x7f1a30138400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40396/0 (socket says 172.21.15.3:40396) 2023-10-30T12:32:33.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.159+0000 7f1a2f7fe640 1 -- 172.21.15.3:0/4231784964 learned_addr learned my addr 172.21.15.3:0/4231784964 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:33.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.159+0000 7f1a2f7fe640 10 monclient(hunting): get_auth_request con 0x7f1a300b1160 auth_method 0 2023-10-30T12:32:33.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.159+0000 7f1a2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:33.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.159+0000 7f1a2f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:33.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.159+0000 7f1a2f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:33.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.159+0000 7f1a34e89640 10 monclient(hunting): get_auth_request con 0x7f1a300af9c0 auth_method 0 2023-10-30T12:32:33.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.159+0000 7f1a34e89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:33.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.159+0000 7f1a34e89640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:33.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.159+0000 7f1a34e89640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:33.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.159+0000 7f1a2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:33.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.159+0000 7f1a2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:33.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.159+0000 7f1a2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:33.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.160+0000 7f1a34e89640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:33.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.160+0000 7f1a34e89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:33.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.160+0000 7f1a34e89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:33.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.161+0000 7f1a2ffff640 1 --2- 172.21.15.3:0/4231784964 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1a180047c0 0x7f1a300af480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.161+0000 7f1a2ffff640 10 monclient(hunting): get_auth_request con 0x7f1a180047c0 auth_method 0 2023-10-30T12:32:33.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.161+0000 7f1a2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:33.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.161+0000 7f1a2ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:33.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.161+0000 7f1a2ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:33.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.161+0000 7f1a2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:33.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.161+0000 7f1a2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:33.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.161+0000 7f1a2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:33.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.162+0000 7f1a34e89640 10 monclient(hunting): handle_auth_done global_id 21814 payload 824 2023-10-30T12:32:33.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.162+0000 7f1a34e89640 1 -- 172.21.15.3:0/4231784964 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1a180047c0 msgr2=0x7f1a300af480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:33.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.162+0000 7f1a34e89640 1 --2- 172.21.15.3:0/4231784964 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1a180047c0 0x7f1a300af480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.162+0000 7f1a34e89640 1 -- 172.21.15.3:0/4231784964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a300b1160 msgr2=0x7f1a30138400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:33.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.162+0000 7f1a34e89640 1 --2- 172.21.15.3:0/4231784964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a300b1160 0x7f1a30138400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.162+0000 7f1a34e89640 10 monclient: _finish_hunting 0 2023-10-30T12:32:33.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.162+0000 7f1a34e89640 1 monclient: found mon.noname-b 2023-10-30T12:32:33.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.162+0000 7f1a34e89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:33.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.162+0000 7f1a34e89640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:33.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.162+0000 7f1a34e89640 1 -- 172.21.15.3:0/4231784964 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a301d6fe0 con 0x7f1a300af9c0 2023-10-30T12:32:33.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.162+0000 7f1a2f7fe640 1 --2- 172.21.15.3:0/4231784964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a300b1160 0x7f1a30138400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:32:33.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.164+0000 7f1a34e89640 1 --2- 172.21.15.3:0/4231784964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a300af9c0 0x7f1a30138940 secure :-1 s=READY pgs=5304 cs=0 l=1 rev1=1 crypto rx=0x7f1a300a9790 tx=0x7f1a2800c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:33.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.164+0000 7f1a2cff9640 10 client.?.objecter ms_handle_connect 0x7f1a300af9c0 2023-10-30T12:32:33.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.164+0000 7f1a2cff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:33.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a2cff9640 1 -- 172.21.15.3:0/4231784964 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a28007070 con 0x7f1a300af9c0 2023-10-30T12:32:33.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:33.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a2cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:33.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a2cff9640 10 monclient: dump: 2023-10-30T12:32:33.184 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:33.184 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:33.185 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.185 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.185 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:33.185 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:33.185 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:33.186 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:33.186 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:33.186 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a2cff9640 10 monclient: _finish_auth 0 2023-10-30T12:32:33.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a2cff9640 10 monclient: _check_auth_tickets 2023-10-30T12:32:33.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:33.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a2cff9640 1 -- 172.21.15.3:0/4231784964 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2800e040 con 0x7f1a300af9c0 2023-10-30T12:32:33.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a2cff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:33.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a2cff9640 1 -- 172.21.15.3:0/4231784964 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a28007440 con 0x7f1a300af9c0 2023-10-30T12:32:33.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:33.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a2cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:33.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a2cff9640 10 monclient: dump: 2023-10-30T12:32:33.188 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:33.188 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:33.188 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.189 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.189 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:33.189 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:33.189 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:33.189 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:33.189 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:33.190 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a368ec640 5 monclient: authenticate success, global_id 21814 2023-10-30T12:32:33.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a368ec640 10 monclient: _renew_subs 2023-10-30T12:32:33.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a368ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:33.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a368ec640 1 -- 172.21.15.3:0/4231784964 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a301d72a0 con 0x7f1a300af9c0 2023-10-30T12:32:33.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a2e7fc640 4 set_mon_vals no callback set 2023-10-30T12:32:33.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a368ec640 10 client.21814.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:33.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a368ec640 10 monclient: _renew_subs 2023-10-30T12:32:33.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a368ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:33.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a368ec640 1 -- 172.21.15.3:0/4231784964 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a301d7740 con 0x7f1a300af9c0 2023-10-30T12:32:33.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.168+0000 7f1a368ec640 1 librados: init done 2023-10-30T12:32:33.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.169+0000 7f1a368ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:33.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.169+0000 7f1a368ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:33.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.169+0000 7f1a368ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:33.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.169+0000 7f1a368ec640 1 -- 172.21.15.3:0/4231784964 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a300aeee0 con 0x7f1a300af9c0 2023-10-30T12:32:33.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.189+0000 7f1a2cff9640 1 -- 172.21.15.3:0/4231784964 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f1a280075b0 con 0x7f1a300af9c0 2023-10-30T12:32:33.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.189+0000 7f1a2cff9640 1 --2- 172.21.15.3:0/4231784964 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1a1c06c7f0 0x7f1a1c06ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.190+0000 7f1a2ffff640 1 --2- 172.21.15.3:0/4231784964 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1a1c06c7f0 0x7f1a1c06ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.190+0000 7f1a2ffff640 10 monclient: get_auth_request con 0x7f1a1c06c7f0 auth_method 0 2023-10-30T12:32:33.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.190+0000 7f1a2cff9640 10 client.21814.objecter ms_handle_connect 0x7f1a1c06c7f0 2023-10-30T12:32:33.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.190+0000 7f1a2ffff640 1 --2- 172.21.15.3:0/4231784964 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1a1c06c7f0 0x7f1a1c06ecb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1a20001af0 tx=0x7f1a20001820 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:33.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.192+0000 7f1a2cff9640 1 -- 172.21.15.3:0/4231784964 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f1a28057af0 con 0x7f1a300af9c0 2023-10-30T12:32:33.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.192+0000 7f1a2cff9640 10 client.21814.objecter ms_dispatch 0x7f1a30000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:33.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.192+0000 7f1a2cff9640 3 client.21814.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:33.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.192+0000 7f1a2cff9640 3 client.21814.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:33.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.192+0000 7f1a2cff9640 20 client.21814.objecter dump_active .. 0 homeless 2023-10-30T12:32:33.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.425+0000 7f1a2cff9640 1 -- 172.21.15.3:0/4231784964 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1a280573f0 con 0x7f1a300af9c0 2023-10-30T12:32:33.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.425+0000 7f1a2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:33.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.425+0000 7f1a2cff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:32:33.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.606+0000 7f1a368ec640 1 -- 172.21.15.3:0/4231784964 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a3006a490 con 0x7f1a1c06c7f0 2023-10-30T12:32:33.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.607+0000 7f1a2cff9640 1 -- 172.21.15.3:0/4231784964 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1a3006a490 con 0x7f1a1c06c7f0 2023-10-30T12:32:33.609 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 0 to deep-scrub 2023-10-30T12:32:33.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 10 librados: watch_flush enter 2023-10-30T12:32:33.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 10 librados: watch_flush exit 2023-10-30T12:32:33.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 20 client.21814.objecter shutdown clearing up homeless session... 2023-10-30T12:32:33.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 10 client.21814.objecter successfully canceled tick 2023-10-30T12:32:33.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 1 -- 172.21.15.3:0/4231784964 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1a1c06c7f0 msgr2=0x7f1a1c06ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:33.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 1 --2- 172.21.15.3:0/4231784964 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1a1c06c7f0 0x7f1a1c06ecb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1a20001af0 tx=0x7f1a20001820 comp rx=0 tx=0).stop 2023-10-30T12:32:33.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 10 monclient: shutdown 2023-10-30T12:32:33.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:33.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 1 -- 172.21.15.3:0/4231784964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a300af9c0 msgr2=0x7f1a30138940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:33.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 1 --2- 172.21.15.3:0/4231784964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a300af9c0 0x7f1a30138940 secure :-1 s=READY pgs=5304 cs=0 l=1 rev1=1 crypto rx=0x7f1a300a9790 tx=0x7f1a2800c7d0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 1 -- 172.21.15.3:0/4231784964 shutdown_connections 2023-10-30T12:32:33.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 1 --2- 172.21.15.3:0/4231784964 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1a1c06c7f0 0x7f1a1c06ecb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 1 --2- 172.21.15.3:0/4231784964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a300af9c0 0x7f1a30138940 unknown :-1 s=CLOSED pgs=5304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 1 --2- 172.21.15.3:0/4231784964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a300b1160 0x7f1a30138400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 1 --2- 172.21.15.3:0/4231784964 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1a180047c0 0x7f1a300af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 1 -- 172.21.15.3:0/4231784964 >> 172.21.15.3:0/4231784964 conn(0x7f1a30065ea0 msgr2=0x7f1a30066280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:33.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 1 -- 172.21.15.3:0/4231784964 shutdown_connections 2023-10-30T12:32:33.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 1 -- 172.21.15.3:0/4231784964 wait complete. 2023-10-30T12:32:33.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.609+0000 7f1a368ec640 1 librados: shutdown 2023-10-30T12:32:33.622 INFO:tasks.ceph:Scrubbing osd.1 2023-10-30T12:32:33.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:32:33.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.730+0000 7f44b0490640 10 monclient: get_monmap_and_config 2023-10-30T12:32:33.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.730+0000 7f44b0490640 10 monclient: build_initial_monmap 2023-10-30T12:32:33.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.731+0000 7f44b0490640 10 monclient: monmap: 2023-10-30T12:32:33.733 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:33.733 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:33.733 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:33.731968+0000 2023-10-30T12:32:33.733 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:33.731968+0000 2023-10-30T12:32:33.734 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:33.734 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:33.734 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:33.734 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:33.734 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:33.735 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:33.735 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:33.735 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.731+0000 7f44b0490640 1 Processor -- start 2023-10-30T12:32:33.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.731+0000 7f44b0490640 1 -- start start 2023-10-30T12:32:33.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.731+0000 7f44b0490640 10 monclient: init 2023-10-30T12:32:33.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.732+0000 7f44b0490640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:33.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.732+0000 7f44b0490640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-10-30T12:32:33.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.732+0000 7f44b0490640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44a80a8850 addr v1:172.21.15.3:6789/0 2023-10-30T12:32:33.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.732+0000 7f44b0490640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44a80b1b90 addr v1:172.21.15.92:6789/0 2023-10-30T12:32:33.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.732+0000 7f44b0490640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f44a80a9350 0x7f44a814d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.732+0000 7f44b0490640 10 monclient(hunting): picked mon.noname-c con 0x7f44a80a9350 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:33.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.732+0000 7f44b0490640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a80aa7b0 con 0x7f44a80a8850 2023-10-30T12:32:33.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.732+0000 7f44b0490640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a80a9780 con 0x7f44a80b1b90 2023-10-30T12:32:33.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.732+0000 7f44b0490640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.732+0000 7f44b0490640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a8074040 con 0x7f44a80a9350 2023-10-30T12:32:33.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.732+0000 7f44b0490640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:33.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.732+0000 7f44b0490640 10 monclient(hunting): authenticate will time out at 2744.946545s 2023-10-30T12:32:33.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.732+0000 7f44ae22c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f44a80a8850 0x7f44a8142380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:48058/0 (socket says 172.21.15.3:48058) 2023-10-30T12:32:33.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.732+0000 7f44ae22c640 1 -- 172.21.15.3:0/551684330 learned_addr learned my addr 172.21.15.3:0/551684330 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:32:33.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.733+0000 7f44aea2d640 1 --2- 172.21.15.3:0/551684330 >> v2:172.21.15.92:3300/0 conn(0x7f44a80a9350 0x7f44a814d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.733+0000 7f44aea2d640 10 monclient(hunting): get_auth_request con 0x7f44a80a9350 auth_method 0 2023-10-30T12:32:33.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.733+0000 7f44aea2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:33.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.733+0000 7f44aea2d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:33.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.733+0000 7f44aea2d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:33.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.733+0000 7f44aea2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:33.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.734+0000 7f44aea2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:33.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.734+0000 7f44aea2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:33.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.735+0000 7f44ad22a640 1 -- 172.21.15.3:0/551684330 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467637557 0 0) 0x7f44a4001110 con 0x7f44a80a8850 2023-10-30T12:32:33.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.735+0000 7f44ad22a640 10 monclient(hunting): _negotiate 2023-10-30T12:32:33.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.735+0000 7f44ad22a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:33.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.735+0000 7f44ad22a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:33.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.735+0000 7f44ad22a640 10 monclient(hunting): my global_id is 16632 2023-10-30T12:32:33.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.735+0000 7f44ad22a640 1 -- 172.21.15.3:0/551684330 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4494003520 con 0x7f44a80a8850 2023-10-30T12:32:33.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.735+0000 7f44aea2d640 10 monclient(hunting): handle_auth_done global_id 21817 payload 274 2023-10-30T12:32:33.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.735+0000 7f44aea2d640 1 -- 172.21.15.3:0/551684330 >> v1:172.21.15.92:6789/0 conn(0x7f44a80b1b90 legacy=0x7f44a813eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:33.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.735+0000 7f44aea2d640 1 -- 172.21.15.3:0/551684330 >> v1:172.21.15.3:6789/0 conn(0x7f44a80a8850 legacy=0x7f44a8142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:33.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.735+0000 7f44aea2d640 10 monclient: _finish_hunting 0 2023-10-30T12:32:33.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.735+0000 7f44aea2d640 1 monclient: found mon.noname-c 2023-10-30T12:32:33.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.735+0000 7f44aea2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:33.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.735+0000 7f44aea2d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:32:33.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.735+0000 7f44aea2d640 1 -- 172.21.15.3:0/551684330 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a8142a90 con 0x7f44a80a9350 2023-10-30T12:32:33.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.738+0000 7f44aea2d640 1 --2- 172.21.15.3:0/551684330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a80a9350 0x7f44a814d3c0 secure :-1 s=READY pgs=5305 cs=0 l=1 rev1=1 crypto rx=0x7f44a0009d90 tx=0x7f44a002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=576bdf34f8680497 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:33.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44ad22a640 1 -- 172.21.15.3:0/551684330 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a002cbd0 con 0x7f44a80a9350 2023-10-30T12:32:33.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44ad22a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:33.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44ad22a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:32:33.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44ad22a640 10 monclient: dump: 2023-10-30T12:32:33.749 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:33.750 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:33.750 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.750 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.750 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:33.750 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:33.750 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:33.751 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:33.751 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:33.751 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44ad22a640 10 monclient: _finish_auth 0 2023-10-30T12:32:33.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44ad22a640 10 monclient: _check_auth_tickets 2023-10-30T12:32:33.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44ad22a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:33.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44ad22a640 1 -- 172.21.15.3:0/551684330 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a002cd40 con 0x7f44a80a9350 2023-10-30T12:32:33.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44ad22a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:33.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44ad22a640 1 -- 172.21.15.3:0/551684330 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0034660 con 0x7f44a80a9350 2023-10-30T12:32:33.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44b0490640 5 monclient: authenticate success, global_id 21817 2023-10-30T12:32:33.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44b0490640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:33.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44b0490640 4 set_mon_vals no callback set 2023-10-30T12:32:33.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44ad22a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:33.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44ad22a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:33.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44ad22a640 10 monclient: dump: 2023-10-30T12:32:33.754 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:33.754 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:33.754 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.754 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.754 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:33.755 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:33.755 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:33.755 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:33.755 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:33.755 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44b0490640 10 monclient: shutdown 2023-10-30T12:32:33.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44b0490640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:33.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44b0490640 1 -- 172.21.15.3:0/551684330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a80a9350 msgr2=0x7f44a814d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:33.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44b0490640 1 --2- 172.21.15.3:0/551684330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a80a9350 0x7f44a814d3c0 secure :-1 s=READY pgs=5305 cs=0 l=1 rev1=1 crypto rx=0x7f44a0009d90 tx=0x7f44a002be20 comp rx=0 tx=0).stop 2023-10-30T12:32:33.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44b0490640 1 -- 172.21.15.3:0/551684330 shutdown_connections 2023-10-30T12:32:33.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44b0490640 1 --2- 172.21.15.3:0/551684330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a80a9350 0x7f44a814d3c0 unknown :-1 s=CLOSED pgs=5305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44b0490640 1 -- 172.21.15.3:0/551684330 >> 172.21.15.3:0/551684330 conn(0x7f44a80b7210 msgr2=0x7f44a80b75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:33.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44b0490640 1 -- 172.21.15.3:0/551684330 shutdown_connections 2023-10-30T12:32:33.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.742+0000 7f44b0490640 1 -- 172.21.15.3:0/551684330 wait complete. 2023-10-30T12:32:33.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.743+0000 7f44b0490640 10 monclient: build_initial_monmap 2023-10-30T12:32:33.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.743+0000 7f44b0490640 10 monclient: monmap: 2023-10-30T12:32:33.758 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:33.758 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:33.758 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:33.758 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:33.759 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:33.759 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:33.759 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:33.759 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:33.759 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:33.759 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.743+0000 7f44b0490640 1 librados: starting msgr at 2023-10-30T12:32:33.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.743+0000 7f44b0490640 1 librados: starting objecter 2023-10-30T12:32:33.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.743+0000 7f44b0490640 1 Processor -- start 2023-10-30T12:32:33.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.743+0000 7f44b0490640 1 -- start start 2023-10-30T12:32:33.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.743+0000 7f44b0490640 1 librados: setting wanted keys 2023-10-30T12:32:33.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.743+0000 7f44b0490640 1 librados: calling monclient init 2023-10-30T12:32:33.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.743+0000 7f44b0490640 10 monclient: init 2023-10-30T12:32:33.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:33.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:32:33.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44a80a8850 0x7f44a8135720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 10 monclient(hunting): picked mon.noname-a con 0x7f44a80a8850 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:33.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f44a80a9350 0x7f44a81da130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 10 monclient(hunting): picked mon.noname-c con 0x7f44a80a9350 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:33.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a80b1b90 0x7f44a80afd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 10 monclient(hunting): picked mon.noname-b con 0x7f44a80b1b90 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:33.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80aa3a0 con 0x7f44a80a8850 2023-10-30T12:32:33.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a8143be0 con 0x7f44a80b1b90 2023-10-30T12:32:33.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80b02a0 con 0x7f44a80a9350 2023-10-30T12:32:33.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:33.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44b0490640 10 monclient(hunting): authenticate will time out at 2744.958575s 2023-10-30T12:32:33.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.744+0000 7f44ae22c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44a80a8850 0x7f44a8135720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44aea2d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a80b1b90 0x7f44a80afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ada2b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f44a80a9350 0x7f44a81da130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ae22c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44a80a8850 0x7f44a8135720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40400/0 (socket says 172.21.15.3:40400) 2023-10-30T12:32:33.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ae22c640 1 -- 172.21.15.3:0/1478140593 learned_addr learned my addr 172.21.15.3:0/1478140593 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:33.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ae22c640 10 monclient(hunting): get_auth_request con 0x7f44a80a8850 auth_method 0 2023-10-30T12:32:33.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ae22c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:33.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ae22c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:33.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ae22c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:33.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ada2b640 10 monclient(hunting): get_auth_request con 0x7f44a80a9350 auth_method 0 2023-10-30T12:32:33.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ada2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:33.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ada2b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:33.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ada2b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:33.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44aea2d640 10 monclient(hunting): get_auth_request con 0x7f44a80b1b90 auth_method 0 2023-10-30T12:32:33.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44aea2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:33.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44aea2d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:33.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44aea2d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:33.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ae22c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:33.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ae22c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:33.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ae22c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:33.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44aea2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:33.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44aea2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:33.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44aea2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:33.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ada2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:33.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ada2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:33.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.745+0000 7f44ada2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:33.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.748+0000 7f44aea2d640 10 monclient(hunting): handle_auth_done global_id 21820 payload 824 2023-10-30T12:32:33.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.748+0000 7f44aea2d640 1 -- 172.21.15.3:0/1478140593 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f44a80a9350 msgr2=0x7f44a81da130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:33.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.748+0000 7f44aea2d640 1 --2- 172.21.15.3:0/1478140593 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f44a80a9350 0x7f44a81da130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.748+0000 7f44aea2d640 1 -- 172.21.15.3:0/1478140593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44a80a8850 msgr2=0x7f44a8135720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:33.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.748+0000 7f44aea2d640 1 --2- 172.21.15.3:0/1478140593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44a80a8850 0x7f44a8135720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.748+0000 7f44aea2d640 10 monclient: _finish_hunting 0 2023-10-30T12:32:33.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.748+0000 7f44aea2d640 1 monclient: found mon.noname-b 2023-10-30T12:32:33.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.748+0000 7f44aea2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:33.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.748+0000 7f44aea2d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:33.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.748+0000 7f44aea2d640 1 -- 172.21.15.3:0/1478140593 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a8143e30 con 0x7f44a80b1b90 2023-10-30T12:32:33.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.750+0000 7f44aea2d640 1 --2- 172.21.15.3:0/1478140593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a80b1b90 0x7f44a80afd60 secure :-1 s=READY pgs=5306 cs=0 l=1 rev1=1 crypto rx=0x7f44a0033000 tx=0x7f44a0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:33.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.750+0000 7f449effd640 10 client.?.objecter ms_handle_connect 0x7f44a80b1b90 2023-10-30T12:32:33.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.750+0000 7f449effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:33.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f449effd640 1 -- 172.21.15.3:0/1478140593 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a003e610 con 0x7f44a80b1b90 2023-10-30T12:32:33.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f449effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:33.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f449effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:33.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f449effd640 10 monclient: dump: 2023-10-30T12:32:33.774 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:33.774 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:33.774 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.775 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.775 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:33.775 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:33.775 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:33.775 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:33.775 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:33.776 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f449effd640 10 monclient: _finish_auth 0 2023-10-30T12:32:33.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f449effd640 10 monclient: _check_auth_tickets 2023-10-30T12:32:33.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f449effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:33.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f449effd640 1 -- 172.21.15.3:0/1478140593 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a003e780 con 0x7f44a80b1b90 2023-10-30T12:32:33.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f449effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:33.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f449effd640 1 -- 172.21.15.3:0/1478140593 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a003eb40 con 0x7f44a80b1b90 2023-10-30T12:32:33.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f449effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:33.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f449effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:33.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f449effd640 10 monclient: dump: 2023-10-30T12:32:33.778 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:33.778 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:33.778 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.778 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.778 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:33.779 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:33.779 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:33.779 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:33.779 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:33.779 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f44b0490640 5 monclient: authenticate success, global_id 21820 2023-10-30T12:32:33.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f44b0490640 10 monclient: _renew_subs 2023-10-30T12:32:33.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f44b0490640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:33.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f44b0490640 1 -- 172.21.15.3:0/1478140593 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a80b06a0 con 0x7f44a80b1b90 2023-10-30T12:32:33.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f44aca29640 4 set_mon_vals no callback set 2023-10-30T12:32:33.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f44b0490640 10 client.21820.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:33.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f44b0490640 10 monclient: _renew_subs 2023-10-30T12:32:33.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f44b0490640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:33.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f44b0490640 1 -- 172.21.15.3:0/1478140593 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a80b0da0 con 0x7f44a80b1b90 2023-10-30T12:32:33.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.754+0000 7f44b0490640 1 librados: init done 2023-10-30T12:32:33.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.755+0000 7f44b0490640 10 client.21820.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:33.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.755+0000 7f44b0490640 15 client.21820.objecter _session_command_op_assign -1 1 2023-10-30T12:32:33.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.755+0000 7f44b0490640 20 client.21820.objecter _get_session osd=-1 returning homeless 2023-10-30T12:32:33.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.755+0000 7f44b0490640 10 client.21820.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:33.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.755+0000 7f44b0490640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:33.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.755+0000 7f44b0490640 1 -- 172.21.15.3:0/1478140593 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f44a8141b80 con 0x7f44a80b1b90 2023-10-30T12:32:33.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.773+0000 7f449effd640 1 -- 172.21.15.3:0/1478140593 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f44a003b050 con 0x7f44a80b1b90 2023-10-30T12:32:33.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.774+0000 7f449effd640 1 --2- 172.21.15.3:0/1478140593 >> v2:172.21.15.92:6800/1750155735 conn(0x7f448c06c7c0 0x7f448c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.774+0000 7f44ae22c640 1 --2- 172.21.15.3:0/1478140593 >> v2:172.21.15.92:6800/1750155735 conn(0x7f448c06c7c0 0x7f448c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.774+0000 7f44ae22c640 10 monclient: get_auth_request con 0x7f448c06c7c0 auth_method 0 2023-10-30T12:32:33.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.774+0000 7f44ae22c640 1 --2- 172.21.15.3:0/1478140593 >> v2:172.21.15.92:6800/1750155735 conn(0x7f448c06c7c0 0x7f448c06ec80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f44a4001380 tx=0x7f44a4007c50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:33.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.774+0000 7f449effd640 10 client.21820.objecter ms_handle_connect 0x7f448c06c7c0 2023-10-30T12:32:33.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 1 -- 172.21.15.3:0/1478140593 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f44a007a810 con 0x7f44a80b1b90 2023-10-30T12:32:33.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 10 client.21820.objecter ms_dispatch 0x7f44a8000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:33.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 3 client.21820.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:33.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 3 client.21820.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:33.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 10 client.21820.objecter checking command 1 2023-10-30T12:32:33.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 1 --2- 172.21.15.3:0/1478140593 >> v2:172.21.15.3:6804/1445650280 conn(0x7f448c073a30 0x7f448c075ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 20 client.21820.objecter _get_session s=0x7f44a003eb40 osd=1 3 2023-10-30T12:32:33.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 20 client.21820.objecter put_session s=0x7f44a003eb40 osd=1 3 2023-10-30T12:32:33.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 15 client.21820.objecter _session_command_op_remove -1 1 2023-10-30T12:32:33.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 20 client.21820.objecter _get_session s=0x7f44a003eb40 osd=1 3 2023-10-30T12:32:33.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 20 client.21820.objecter get_session s=0x7f44a003eb40 osd=1 3 2023-10-30T12:32:33.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 15 client.21820.objecter _session_command_op_assign 1 1 2023-10-30T12:32:33.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 20 client.21820.objecter put_session s=0x7f44a003eb40 osd=1 4 2023-10-30T12:32:33.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 10 client.21820.objecter _send_command 1 2023-10-30T12:32:33.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 1 -- 172.21.15.3:0/1478140593 --> v2:172.21.15.3:6804/1445650280 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f44a003e610 con 0x7f448c073a30 2023-10-30T12:32:33.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.776+0000 7f449effd640 20 client.21820.objecter dump_active .. 0 homeless 2023-10-30T12:32:33.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.777+0000 7f44ada2b640 1 --2- 172.21.15.3:0/1478140593 >> v2:172.21.15.3:6804/1445650280 conn(0x7f448c073a30 0x7f448c075ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.777+0000 7f449effd640 1 -- 172.21.15.3:0/1478140593 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44a0009a70 con 0x7f44a80b1b90 2023-10-30T12:32:33.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.777+0000 7f449effd640 10 monclient: handle_get_version_reply finishing 1 version 900 2023-10-30T12:32:33.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.777+0000 7f44ada2b640 10 monclient: get_auth_request con 0x7f448c073a30 auth_method 0 2023-10-30T12:32:33.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.780+0000 7f44ada2b640 1 --2- 172.21.15.3:0/1478140593 >> v2:172.21.15.3:6804/1445650280 conn(0x7f448c073a30 0x7f448c075ed0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:33.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.780+0000 7f449effd640 10 client.21820.objecter ms_handle_connect 0x7f448c073a30 2023-10-30T12:32:33.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.807+0000 7f449effd640 1 -- 172.21.15.3:0/1478140593 <== osd.1 v2:172.21.15.3:6804/1445650280 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f44a003e610 con 0x7f448c073a30 2023-10-30T12:32:33.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.807+0000 7f449effd640 10 client.21820.objecter ms_dispatch 0x7f44a8000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:32:33.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.807+0000 7f449effd640 10 client.21820.objecter _finish_command 1 = system:0 2023-10-30T12:32:33.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.807+0000 7f449effd640 20 client.21820.objecter put_session s=0x7f44a003eb40 osd=1 4 2023-10-30T12:32:33.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.807+0000 7f449effd640 15 client.21820.objecter _session_command_op_remove 1 1 2023-10-30T12:32:33.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.820+0000 7f44b0490640 10 client.21820.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:32:33.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.820+0000 7f44b0490640 15 client.21820.objecter _session_command_op_assign -1 2 2023-10-30T12:32:33.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.820+0000 7f44b0490640 20 client.21820.objecter _get_session s=0x7f44a003eb40 osd=1 3 2023-10-30T12:32:33.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.820+0000 7f44b0490640 20 client.21820.objecter put_session s=0x7f44a003eb40 osd=1 3 2023-10-30T12:32:33.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.820+0000 7f44b0490640 20 client.21820.objecter _get_session s=0x7f44a003eb40 osd=1 3 2023-10-30T12:32:33.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.820+0000 7f44b0490640 15 client.21820.objecter _session_command_op_remove -1 2 2023-10-30T12:32:33.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.820+0000 7f44b0490640 20 client.21820.objecter get_session s=0x7f44a003eb40 osd=1 3 2023-10-30T12:32:33.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.820+0000 7f44b0490640 15 client.21820.objecter _session_command_op_assign 1 2 2023-10-30T12:32:33.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.820+0000 7f44b0490640 20 client.21820.objecter put_session s=0x7f44a003eb40 osd=1 4 2023-10-30T12:32:33.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.820+0000 7f44b0490640 10 client.21820.objecter _send_command 2 2023-10-30T12:32:33.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.820+0000 7f44b0490640 1 -- 172.21.15.3:0/1478140593 --> v2:172.21.15.3:6804/1445650280 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f44a80b2000 con 0x7f448c073a30 2023-10-30T12:32:33.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.836+0000 7f449effd640 1 -- 172.21.15.3:0/1478140593 <== osd.1 v2:172.21.15.3:6804/1445650280 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4498009550 con 0x7f448c073a30 2023-10-30T12:32:33.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.836+0000 7f449effd640 10 client.21820.objecter ms_dispatch 0x7f44a8000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:32:33.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.836+0000 7f449effd640 10 client.21820.objecter _finish_command 2 = system:0 2023-10-30T12:32:33.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.836+0000 7f449effd640 20 client.21820.objecter put_session s=0x7f44a003eb40 osd=1 4 2023-10-30T12:32:33.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.836+0000 7f449effd640 15 client.21820.objecter _session_command_op_remove 1 2 2023-10-30T12:32:33.839 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-10-30T12:32:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:32:33.839 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-10-30T12:32:33.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 10 librados: watch_flush enter 2023-10-30T12:32:33.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 10 librados: watch_flush exit 2023-10-30T12:32:33.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 10 client.21820.objecter close_session for osd.1 2023-10-30T12:32:33.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 1 -- 172.21.15.3:0/1478140593 >> v2:172.21.15.3:6804/1445650280 conn(0x7f448c073a30 msgr2=0x7f448c075ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:33.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 1 --2- 172.21.15.3:0/1478140593 >> v2:172.21.15.3:6804/1445650280 conn(0x7f448c073a30 0x7f448c075ed0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 20 client.21820.objecter put_session s=0x7f44a003eb40 osd=1 1 2023-10-30T12:32:33.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 20 client.21820.objecter shutdown clearing up homeless session... 2023-10-30T12:32:33.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 10 client.21820.objecter successfully canceled tick 2023-10-30T12:32:33.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 1 -- 172.21.15.3:0/1478140593 >> v2:172.21.15.92:6800/1750155735 conn(0x7f448c06c7c0 msgr2=0x7f448c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:33.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 1 --2- 172.21.15.3:0/1478140593 >> v2:172.21.15.92:6800/1750155735 conn(0x7f448c06c7c0 0x7f448c06ec80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f44a4001380 tx=0x7f44a4007c50 comp rx=0 tx=0).stop 2023-10-30T12:32:33.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 10 monclient: shutdown 2023-10-30T12:32:33.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:33.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 1 -- 172.21.15.3:0/1478140593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a80b1b90 msgr2=0x7f44a80afd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:33.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 1 --2- 172.21.15.3:0/1478140593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a80b1b90 0x7f44a80afd60 secure :-1 s=READY pgs=5306 cs=0 l=1 rev1=1 crypto rx=0x7f44a0033000 tx=0x7f44a0004a40 comp rx=0 tx=0).stop 2023-10-30T12:32:33.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44aea2d640 1 -- 172.21.15.3:0/1478140593 reap_dead start 2023-10-30T12:32:33.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 1 -- 172.21.15.3:0/1478140593 shutdown_connections 2023-10-30T12:32:33.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 1 -- 172.21.15.3:0/1478140593 >> 172.21.15.3:0/1478140593 conn(0x7f44a80b7210 msgr2=0x7f44a8142380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:33.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 1 -- 172.21.15.3:0/1478140593 shutdown_connections 2023-10-30T12:32:33.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 1 -- 172.21.15.3:0/1478140593 wait complete. 2023-10-30T12:32:33.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.837+0000 7f44b0490640 1 librados: shutdown 2023-10-30T12:32:33.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-30T12:32:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.956+0000 7ff718b7c640 10 monclient: get_monmap_and_config 2023-10-30T12:32:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.956+0000 7ff718b7c640 10 monclient: build_initial_monmap 2023-10-30T12:32:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.956+0000 7ff718b7c640 10 monclient: monmap: 2023-10-30T12:32:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:33.957702+0000 2023-10-30T12:32:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:33.957702+0000 2023-10-30T12:32:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:33.961 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.957+0000 7ff718b7c640 1 Processor -- start 2023-10-30T12:32:33.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.957+0000 7ff718b7c640 1 -- start start 2023-10-30T12:32:33.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.957+0000 7ff718b7c640 10 monclient: init 2023-10-30T12:32:33.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.958+0000 7ff718b7c640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:33.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.958+0000 7ff718b7c640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-10-30T12:32:33.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.958+0000 7ff718b7c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7140b4350 0x7ff7140a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.958+0000 7ff718b7c640 10 monclient(hunting): picked mon.noname-b con 0x7ff7140b4350 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:33.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.958+0000 7ff718b7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff71413eb50 addr v1:172.21.15.3:6789/0 2023-10-30T12:32:33.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.958+0000 7ff718b7c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff714144f50 0x7ff7141453e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.958+0000 7ff718b7c640 10 monclient(hunting): picked mon.noname-c con 0x7ff714144f50 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:33.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.958+0000 7ff718b7c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7140aa6e0 con 0x7ff71413eb50 2023-10-30T12:32:33.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.958+0000 7ff718b7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.958+0000 7ff718b7c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7140679f0 con 0x7ff714144f50 2023-10-30T12:32:33.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.958+0000 7ff718b7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.958+0000 7ff718b7c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7140b4730 con 0x7ff7140b4350 2023-10-30T12:32:33.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.958+0000 7ff718b7c640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:33.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.958+0000 7ff718b7c640 10 monclient(hunting): authenticate will time out at 2745.172440s 2023-10-30T12:32:33.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.958+0000 7ff71259c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7140b4350 0x7ff7140a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff70bfff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff71413eb50 0x7ff71413ef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:48066/0 (socket says 172.21.15.3:48066) 2023-10-30T12:32:33.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff70bfff640 1 -- 172.21.15.3:0/408452526 learned_addr learned my addr 172.21.15.3:0/408452526 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:32:33.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff712d9d640 1 --2- 172.21.15.3:0/408452526 >> v2:172.21.15.92:3300/0 conn(0x7ff714144f50 0x7ff7141453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff71259c640 10 monclient(hunting): get_auth_request con 0x7ff7140b4350 auth_method 0 2023-10-30T12:32:33.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff71259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:33.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff71259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:33.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff71259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:33.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff712d9d640 10 monclient(hunting): get_auth_request con 0x7ff714144f50 auth_method 0 2023-10-30T12:32:33.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff712d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:33.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff712d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:33.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff712d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:33.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff712d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:33.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff712d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:33.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff712d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:33.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff71259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:33.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff71259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:33.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.959+0000 7ff71259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:33.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.961+0000 7ff712d9d640 10 monclient(hunting): handle_auth_done global_id 21823 payload 274 2023-10-30T12:32:33.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.961+0000 7ff712d9d640 1 -- 172.21.15.3:0/408452526 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7140b4350 msgr2=0x7ff7140a83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:33.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.961+0000 7ff712d9d640 1 --2- 172.21.15.3:0/408452526 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7140b4350 0x7ff7140a83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.961+0000 7ff712d9d640 1 -- 172.21.15.3:0/408452526 >> v1:172.21.15.3:6789/0 conn(0x7ff71413eb50 legacy=0x7ff71413ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:33.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.961+0000 7ff712d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:32:33.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.961+0000 7ff712d9d640 1 monclient: found mon.noname-c 2023-10-30T12:32:33.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.961+0000 7ff712d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:33.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.961+0000 7ff712d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:32:33.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.961+0000 7ff712d9d640 1 -- 172.21.15.3:0/408452526 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71413f680 con 0x7ff714144f50 2023-10-30T12:32:33.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.963+0000 7ff712d9d640 1 --2- 172.21.15.3:0/408452526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff714144f50 0x7ff7141453e0 secure :-1 s=READY pgs=5307 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009d90 tx=0x7ff70c02bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b88324a671c6ffe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:33.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff711d9b640 1 -- 172.21.15.3:0/408452526 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c02cb30 con 0x7ff714144f50 2023-10-30T12:32:33.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff711d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:33.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff711d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:32:33.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff711d9b640 10 monclient: dump: 2023-10-30T12:32:33.980 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:33.980 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:33.980 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.981 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.981 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:33.981 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:33.981 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:33.981 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:33.982 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:33.982 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff711d9b640 10 monclient: _finish_auth 0 2023-10-30T12:32:33.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff711d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:32:33.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff711d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:33.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff711d9b640 1 -- 172.21.15.3:0/408452526 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70c02cca0 con 0x7ff714144f50 2023-10-30T12:32:33.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff711d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:33.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff711d9b640 1 -- 172.21.15.3:0/408452526 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c034660 con 0x7ff714144f50 2023-10-30T12:32:33.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff711d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:33.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff711d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:33.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff711d9b640 10 monclient: dump: 2023-10-30T12:32:33.984 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:33.984 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:33.985 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.985 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:33.985 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:33.985 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:33.986 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:33.986 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:33.987 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:33.987 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff718b7c640 5 monclient: authenticate success, global_id 21823 2023-10-30T12:32:33.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff718b7c640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:33.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.967+0000 7ff718b7c640 4 set_mon_vals no callback set 2023-10-30T12:32:33.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.968+0000 7ff718b7c640 10 monclient: shutdown 2023-10-30T12:32:33.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.968+0000 7ff718b7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:33.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.968+0000 7ff718b7c640 1 -- 172.21.15.3:0/408452526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff714144f50 msgr2=0x7ff7141453e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:33.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.968+0000 7ff718b7c640 1 --2- 172.21.15.3:0/408452526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff714144f50 0x7ff7141453e0 secure :-1 s=READY pgs=5307 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009d90 tx=0x7ff70c02bd80 comp rx=0 tx=0).stop 2023-10-30T12:32:33.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.968+0000 7ff718b7c640 1 -- 172.21.15.3:0/408452526 shutdown_connections 2023-10-30T12:32:33.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.968+0000 7ff718b7c640 1 --2- 172.21.15.3:0/408452526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff714144f50 0x7ff7141453e0 unknown :-1 s=CLOSED pgs=5307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.968+0000 7ff718b7c640 1 --2- 172.21.15.3:0/408452526 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7140b4350 0x7ff7140a83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:33.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.968+0000 7ff718b7c640 1 -- 172.21.15.3:0/408452526 >> 172.21.15.3:0/408452526 conn(0x7ff714065ea0 msgr2=0x7ff7140662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:33.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.968+0000 7ff718b7c640 1 -- 172.21.15.3:0/408452526 shutdown_connections 2023-10-30T12:32:33.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.968+0000 7ff718b7c640 1 -- 172.21.15.3:0/408452526 wait complete. 2023-10-30T12:32:33.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.969+0000 7ff718b7c640 10 monclient: build_initial_monmap 2023-10-30T12:32:33.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.969+0000 7ff718b7c640 10 monclient: monmap: 2023-10-30T12:32:33.991 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:33.991 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:33.991 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:33.992 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:33.992 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:33.992 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:33.992 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:33.992 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:33.993 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:33.993 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:33.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.969+0000 7ff718b7c640 1 librados: starting msgr at 2023-10-30T12:32:33.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.969+0000 7ff718b7c640 1 librados: starting objecter 2023-10-30T12:32:33.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.969+0000 7ff718b7c640 1 Processor -- start 2023-10-30T12:32:33.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.969+0000 7ff718b7c640 1 -- start start 2023-10-30T12:32:33.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.969+0000 7ff718b7c640 1 librados: setting wanted keys 2023-10-30T12:32:33.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.969+0000 7ff718b7c640 1 librados: calling monclient init 2023-10-30T12:32:33.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.969+0000 7ff718b7c640 10 monclient: init 2023-10-30T12:32:33.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:33.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:32:33.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7140b4350 0x7ff7140b1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 10 monclient(hunting): picked mon.noname-c con 0x7ff7140b4350 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:33.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff71413eb50 0x7ff7140b5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 10 monclient(hunting): picked mon.noname-a con 0x7ff71413eb50 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:33.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff714144f50 0x7ff7140af190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:33.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 10 monclient(hunting): picked mon.noname-b con 0x7ff714144f50 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:33.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff714145480 con 0x7ff71413eb50 2023-10-30T12:32:33.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7140b5cb0 con 0x7ff714144f50 2023-10-30T12:32:33.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:33.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7140aec70 con 0x7ff7140b4350 2023-10-30T12:32:33.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:33.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff718b7c640 10 monclient(hunting): authenticate will time out at 2745.184407s 2023-10-30T12:32:33.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff70bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff71413eb50 0x7ff7140b5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff71259c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7140b4350 0x7ff7140b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.970+0000 7ff712d9d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff714144f50 0x7ff7140af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:33.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff70bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff71413eb50 0x7ff7140b5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40416/0 (socket says 172.21.15.3:40416) 2023-10-30T12:32:33.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff71259c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7140b4350 0x7ff7140b1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:37592/0 (socket says 172.21.15.3:37592) 2023-10-30T12:32:33.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff70bfff640 1 -- 172.21.15.3:0/3357311192 learned_addr learned my addr 172.21.15.3:0/3357311192 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:34.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff70bfff640 10 monclient(hunting): get_auth_request con 0x7ff71413eb50 auth_method 0 2023-10-30T12:32:34.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff70bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:34.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff70bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:34.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff70bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:34.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff71259c640 10 monclient(hunting): get_auth_request con 0x7ff7140b4350 auth_method 0 2023-10-30T12:32:34.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff71259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:34.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff71259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:34.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff71259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:34.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff712d9d640 10 monclient(hunting): get_auth_request con 0x7ff714144f50 auth_method 0 2023-10-30T12:32:34.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff712d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:34.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff712d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:34.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff712d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:34.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff71259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:34.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff71259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:34.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff71259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:34.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff70bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:34.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff70bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:34.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.971+0000 7ff70bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:34.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.972+0000 7ff712d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:34.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.972+0000 7ff712d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:34.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.972+0000 7ff712d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:34.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.974+0000 7ff71259c640 10 monclient(hunting): handle_auth_done global_id 15563 payload 824 2023-10-30T12:32:34.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.974+0000 7ff71259c640 1 -- 172.21.15.3:0/3357311192 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff714144f50 msgr2=0x7ff7140af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.974+0000 7ff71259c640 1 --2- 172.21.15.3:0/3357311192 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff714144f50 0x7ff7140af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.974+0000 7ff71259c640 1 -- 172.21.15.3:0/3357311192 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff71413eb50 msgr2=0x7ff7140b5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.974+0000 7ff71259c640 1 --2- 172.21.15.3:0/3357311192 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff71413eb50 0x7ff7140b5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.974+0000 7ff71259c640 10 monclient: _finish_hunting 0 2023-10-30T12:32:34.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.974+0000 7ff71259c640 1 monclient: found mon.noname-c 2023-10-30T12:32:34.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.974+0000 7ff71259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:34.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.974+0000 7ff71259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.3:3301/0 2023-10-30T12:32:34.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.974+0000 7ff71259c640 1 -- 172.21.15.3:0/3357311192 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7140b7620 con 0x7ff7140b4350 2023-10-30T12:32:34.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.974+0000 7ff70bfff640 1 --2- 172.21.15.3:0/3357311192 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff71413eb50 0x7ff7140b5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:32:34.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.976+0000 7ff71259c640 1 --2- 172.21.15.3:0/3357311192 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7140b4350 0x7ff7140b1080 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7ff70400e850 tx=0x7ff70400ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:34.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.976+0000 7ff70affd640 10 client.?.objecter ms_handle_connect 0x7ff7140b4350 2023-10-30T12:32:34.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.976+0000 7ff70affd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:34.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff70affd640 1 -- 172.21.15.3:0/3357311192 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70400c940 con 0x7ff7140b4350 2023-10-30T12:32:34.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff70affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:34.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff70affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:32:34.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff70affd640 10 monclient: dump: 2023-10-30T12:32:34.010 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:34.010 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:34.010 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.010 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.011 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:34.011 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:34.011 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:34.011 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:34.011 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:34.012 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:34.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff70affd640 10 monclient: _finish_auth 0 2023-10-30T12:32:34.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff70affd640 10 monclient: _check_auth_tickets 2023-10-30T12:32:34.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff70affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:34.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff70affd640 1 -- 172.21.15.3:0/3357311192 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70400d040 con 0x7ff7140b4350 2023-10-30T12:32:34.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff70affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:34.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff70affd640 1 -- 172.21.15.3:0/3357311192 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70400ccd0 con 0x7ff7140b4350 2023-10-30T12:32:34.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff70affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:34.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff70affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:32:34.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff70affd640 10 monclient: dump: 2023-10-30T12:32:34.014 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:34.014 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:34.014 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.015 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.015 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:34.015 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:34.015 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:34.015 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:34.016 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:34.016 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:34.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff718b7c640 5 monclient: authenticate success, global_id 15563 2023-10-30T12:32:34.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff718b7c640 10 monclient: _renew_subs 2023-10-30T12:32:34.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff718b7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:32:34.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff718b7c640 1 -- 172.21.15.3:0/3357311192 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7140af6d0 con 0x7ff7140b4350 2023-10-30T12:32:34.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff71159a640 4 set_mon_vals no callback set 2023-10-30T12:32:34.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff718b7c640 10 client.15563.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:34.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff718b7c640 10 monclient: _renew_subs 2023-10-30T12:32:34.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff718b7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:32:34.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff718b7c640 1 -- 172.21.15.3:0/3357311192 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7140ae400 con 0x7ff7140b4350 2023-10-30T12:32:34.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.980+0000 7ff718b7c640 1 librados: init done 2023-10-30T12:32:34.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.981+0000 7ff718b7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:34.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.981+0000 7ff718b7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:34.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.981+0000 7ff718b7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:32:34.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:33.981+0000 7ff718b7c640 1 -- 172.21.15.3:0/3357311192 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7140af8d0 con 0x7ff7140b4350 2023-10-30T12:32:34.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.000+0000 7ff70affd640 1 -- 172.21.15.3:0/3357311192 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7ff704013470 con 0x7ff7140b4350 2023-10-30T12:32:34.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.001+0000 7ff70affd640 1 --2- 172.21.15.3:0/3357311192 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff6f406c810 0x7ff6f406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.001+0000 7ff70bfff640 1 --2- 172.21.15.3:0/3357311192 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff6f406c810 0x7ff6f406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.001+0000 7ff70bfff640 10 monclient: get_auth_request con 0x7ff6f406c810 auth_method 0 2023-10-30T12:32:34.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.001+0000 7ff70bfff640 1 --2- 172.21.15.3:0/3357311192 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff6f406c810 0x7ff6f406ecd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff7140b7130 tx=0x7ff6fc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:34.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.001+0000 7ff70affd640 10 client.15563.objecter ms_handle_connect 0x7ff6f406c810 2023-10-30T12:32:34.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.252+0000 7ff70affd640 1 -- 172.21.15.3:0/3357311192 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7ff704058670 con 0x7ff7140b4350 2023-10-30T12:32:34.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.252+0000 7ff70affd640 10 client.15563.objecter ms_dispatch 0x7ff714000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:34.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.252+0000 7ff70affd640 3 client.15563.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:34.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.252+0000 7ff70affd640 3 client.15563.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:34.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.252+0000 7ff70affd640 20 client.15563.objecter dump_active .. 0 homeless 2023-10-30T12:32:34.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.252+0000 7ff70affd640 1 -- 172.21.15.3:0/3357311192 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff70400ce40 con 0x7ff7140b4350 2023-10-30T12:32:34.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.252+0000 7ff70affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:34.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.252+0000 7ff70affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:32:34.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.437+0000 7ff718b7c640 1 -- 172.21.15.3:0/3357311192 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7140a8440 con 0x7ff6f406c810 2023-10-30T12:32:34.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.437+0000 7ff70affd640 1 -- 172.21.15.3:0/3357311192 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff7140a8440 con 0x7ff6f406c810 2023-10-30T12:32:34.440 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 1 to deep-scrub 2023-10-30T12:32:34.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 10 librados: watch_flush enter 2023-10-30T12:32:34.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 10 librados: watch_flush exit 2023-10-30T12:32:34.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 20 client.15563.objecter shutdown clearing up homeless session... 2023-10-30T12:32:34.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 10 client.15563.objecter successfully canceled tick 2023-10-30T12:32:34.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 1 -- 172.21.15.3:0/3357311192 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff6f406c810 msgr2=0x7ff6f406ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 1 --2- 172.21.15.3:0/3357311192 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff6f406c810 0x7ff6f406ecd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff7140b7130 tx=0x7ff6fc009040 comp rx=0 tx=0).stop 2023-10-30T12:32:34.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 10 monclient: shutdown 2023-10-30T12:32:34.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:34.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 1 -- 172.21.15.3:0/3357311192 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7140b4350 msgr2=0x7ff7140b1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 1 --2- 172.21.15.3:0/3357311192 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7140b4350 0x7ff7140b1080 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7ff70400e850 tx=0x7ff70400ed10 comp rx=0 tx=0).stop 2023-10-30T12:32:34.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 1 -- 172.21.15.3:0/3357311192 shutdown_connections 2023-10-30T12:32:34.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 1 --2- 172.21.15.3:0/3357311192 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff6f406c810 0x7ff6f406ecd0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 1 --2- 172.21.15.3:0/3357311192 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff714144f50 0x7ff7140af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 1 --2- 172.21.15.3:0/3357311192 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff71413eb50 0x7ff7140b5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 1 --2- 172.21.15.3:0/3357311192 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7140b4350 0x7ff7140b1080 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 1 -- 172.21.15.3:0/3357311192 >> 172.21.15.3:0/3357311192 conn(0x7ff714065ea0 msgr2=0x7ff7140b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:34.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 1 -- 172.21.15.3:0/3357311192 shutdown_connections 2023-10-30T12:32:34.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 1 -- 172.21.15.3:0/3357311192 wait complete. 2023-10-30T12:32:34.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.440+0000 7ff718b7c640 1 librados: shutdown 2023-10-30T12:32:34.454 INFO:tasks.ceph:Scrubbing osd.2 2023-10-30T12:32:34.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:32:34.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.564+0000 7f0bfbe44640 10 monclient: get_monmap_and_config 2023-10-30T12:32:34.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.564+0000 7f0bfbe44640 10 monclient: build_initial_monmap 2023-10-30T12:32:34.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.564+0000 7f0bfbe44640 10 monclient: monmap: 2023-10-30T12:32:34.566 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:34.566 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:34.566 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:34.565102+0000 2023-10-30T12:32:34.567 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:34.565102+0000 2023-10-30T12:32:34.567 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:34.567 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:34.567 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:34.568 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:34.568 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:34.568 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:34.568 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:34.568 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:34.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.564+0000 7f0bfbe44640 1 Processor -- start 2023-10-30T12:32:34.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.564+0000 7f0bfbe44640 1 -- start start 2023-10-30T12:32:34.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.564+0000 7f0bfbe44640 10 monclient: init 2023-10-30T12:32:34.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:34.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-10-30T12:32:34.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0bf40b4420 0x7f0bf413ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 10 monclient(hunting): picked mon.noname-c con 0x7f0bf40b4420 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:34.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf40b12d0 0x7f0bf413ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 10 monclient(hunting): picked mon.noname-b con 0x7f0bf40b12d0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:34.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0bf4141910 0x7f0bf4141da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 10 monclient(hunting): picked mon.noname-a con 0x7f0bf4141910 addr v2:172.21.15.3:3300/0 2023-10-30T12:32:34.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:34.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf4067a10 con 0x7f0bf4141910 2023-10-30T12:32:34.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:34.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf40b4800 con 0x7f0bf40b4420 2023-10-30T12:32:34.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:34.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40a9120 con 0x7f0bf40b12d0 2023-10-30T12:32:34.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:34.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.565+0000 7f0bfbe44640 10 monclient(hunting): authenticate will time out at 2745.779620s 2023-10-30T12:32:34.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bfa3e1640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0bf4141910 0x7f0bf4141da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bf93df640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf40b12d0 0x7f0bf413ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bfa3e1640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0bf4141910 0x7f0bf4141da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40420/0 (socket says 172.21.15.3:40420) 2023-10-30T12:32:34.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bfa3e1640 1 -- 172.21.15.3:0/72699457 learned_addr learned my addr 172.21.15.3:0/72699457 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:34.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bfa3e1640 10 monclient(hunting): get_auth_request con 0x7f0bf4141910 auth_method 0 2023-10-30T12:32:34.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bfa3e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:34.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bfa3e1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:34.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bfa3e1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:34.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bf93df640 10 monclient(hunting): get_auth_request con 0x7f0bf40b12d0 auth_method 0 2023-10-30T12:32:34.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bf93df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:34.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bf93df640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:34.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bf93df640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:34.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bf9be0640 1 --2- 172.21.15.3:0/72699457 >> v2:172.21.15.92:3300/0 conn(0x7f0bf40b4420 0x7f0bf413ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bf9be0640 10 monclient(hunting): get_auth_request con 0x7f0bf40b4420 auth_method 0 2023-10-30T12:32:34.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bf9be0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bf9be0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bf9be0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bfa3e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bfa3e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bfa3e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bf93df640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bf93df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.566+0000 7f0bf93df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.567+0000 7f0bf9be0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:34.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.567+0000 7f0bf9be0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:34.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.567+0000 7f0bf9be0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:34.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.568+0000 7f0bfa3e1640 10 monclient(hunting): handle_auth_done global_id 16644 payload 274 2023-10-30T12:32:34.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.568+0000 7f0bfa3e1640 1 -- 172.21.15.3:0/72699457 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf40b12d0 msgr2=0x7f0bf413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:34.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.568+0000 7f0bfa3e1640 1 --2- 172.21.15.3:0/72699457 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf40b12d0 0x7f0bf413ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.568+0000 7f0bfa3e1640 1 -- 172.21.15.3:0/72699457 >> v2:172.21.15.92:3300/0 conn(0x7f0bf40b4420 msgr2=0x7f0bf413ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:34.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.568+0000 7f0bfa3e1640 1 --2- 172.21.15.3:0/72699457 >> v2:172.21.15.92:3300/0 conn(0x7f0bf40b4420 0x7f0bf413ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.568+0000 7f0bfa3e1640 10 monclient: _finish_hunting 0 2023-10-30T12:32:34.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.568+0000 7f0bfa3e1640 1 monclient: found mon.noname-a 2023-10-30T12:32:34.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.568+0000 7f0bfa3e1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:34.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.568+0000 7f0bfa3e1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:32:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.568+0000 7f0bfa3e1640 1 -- 172.21.15.3:0/72699457 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf40aa460 con 0x7f0bf4141910 2023-10-30T12:32:34.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.570+0000 7f0bfa3e1640 1 --2- 172.21.15.3:0/72699457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf4141910 0x7f0bf4141da0 secure :-1 s=READY pgs=3608 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40b4180 tx=0x7f0bec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc12bb4b73459b06 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:34.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bf8bde640 1 -- 172.21.15.3:0/72699457 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec0317e0 con 0x7f0bf4141910 2023-10-30T12:32:34.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bf8bde640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:34.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bf8bde640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:32:34.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bf8bde640 10 monclient: dump: 2023-10-30T12:32:34.587 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:34.587 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:34.587 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.588 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.588 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:34.588 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:34.588 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:34.588 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:34.589 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:34.589 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:34.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bf8bde640 10 monclient: _finish_auth 0 2023-10-30T12:32:34.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bf8bde640 10 monclient: _check_auth_tickets 2023-10-30T12:32:34.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bf8bde640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:34.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bf8bde640 1 -- 172.21.15.3:0/72699457 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec031950 con 0x7f0bf4141910 2023-10-30T12:32:34.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bfbe44640 5 monclient: authenticate success, global_id 16644 2023-10-30T12:32:34.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bfbe44640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:32:34.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bf8bde640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:34.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bfbe44640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:34.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bfbe44640 4 set_mon_vals no callback set 2023-10-30T12:32:34.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bf8bde640 1 -- 172.21.15.3:0/72699457 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec031cf0 con 0x7f0bf4141910 2023-10-30T12:32:34.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bf8bde640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:34.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bf8bde640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:32:34.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bf8bde640 10 monclient: dump: 2023-10-30T12:32:34.592 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:34.592 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:34.592 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.592 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.592 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:34.592 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:34.593 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:34.593 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:34.593 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:34.593 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:34.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bfbe44640 10 monclient: shutdown 2023-10-30T12:32:34.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bfbe44640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:34.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bfbe44640 1 -- 172.21.15.3:0/72699457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf4141910 msgr2=0x7f0bf4141da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bfbe44640 1 --2- 172.21.15.3:0/72699457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf4141910 0x7f0bf4141da0 secure :-1 s=READY pgs=3608 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40b4180 tx=0x7f0bec02b880 comp rx=0 tx=0).stop 2023-10-30T12:32:34.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bfbe44640 1 -- 172.21.15.3:0/72699457 shutdown_connections 2023-10-30T12:32:34.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bfbe44640 1 --2- 172.21.15.3:0/72699457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf4141910 0x7f0bf4141da0 unknown :-1 s=CLOSED pgs=3608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bfbe44640 1 --2- 172.21.15.3:0/72699457 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf40b12d0 0x7f0bf413ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bfbe44640 1 --2- 172.21.15.3:0/72699457 >> v2:172.21.15.92:3300/0 conn(0x7f0bf40b4420 0x7f0bf413ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bfbe44640 1 -- 172.21.15.3:0/72699457 >> 172.21.15.3:0/72699457 conn(0x7f0bf4065ea0 msgr2=0x7f0bf40662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:34.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.574+0000 7f0bfbe44640 1 -- 172.21.15.3:0/72699457 shutdown_connections 2023-10-30T12:32:34.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.575+0000 7f0bfbe44640 1 -- 172.21.15.3:0/72699457 wait complete. 2023-10-30T12:32:34.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.575+0000 7f0bfbe44640 10 monclient: build_initial_monmap 2023-10-30T12:32:34.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.575+0000 7f0bfbe44640 10 monclient: monmap: 2023-10-30T12:32:34.596 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:34.596 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:34.596 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:34.597 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:34.597 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:34.597 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:34.597 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:34.597 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:34.598 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:34.598 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:34.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.575+0000 7f0bfbe44640 1 librados: starting msgr at 2023-10-30T12:32:34.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.575+0000 7f0bfbe44640 1 librados: starting objecter 2023-10-30T12:32:34.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 1 Processor -- start 2023-10-30T12:32:34.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 1 -- start start 2023-10-30T12:32:34.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 1 librados: setting wanted keys 2023-10-30T12:32:34.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 1 librados: calling monclient init 2023-10-30T12:32:34.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 10 monclient: init 2023-10-30T12:32:34.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:34.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:32:34.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf40b12d0 0x7f0bf41cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 10 monclient(hunting): picked mon.noname-c con 0x7f0bf40b12d0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:34.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf40b4420 0x7f0bf41c7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 10 monclient(hunting): picked mon.noname-a con 0x7f0bf40b4420 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:34.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bf4141910 0x7f0bf41d6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 10 monclient(hunting): picked mon.noname-b con 0x7f0bf4141910 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:34.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:34.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40a83d0 con 0x7f0bf40b4420 2023-10-30T12:32:34.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:34.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf41d6a20 con 0x7f0bf4141910 2023-10-30T12:32:34.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:34.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf41ca060 con 0x7f0bf40b12d0 2023-10-30T12:32:34.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:34.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.576+0000 7f0bfbe44640 10 monclient(hunting): authenticate will time out at 2745.790697s 2023-10-30T12:32:34.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf93df640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf40b4420 0x7f0bf41c7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf9be0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf40b12d0 0x7f0bf41cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bfa3e1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bf4141910 0x7f0bf41d6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf93df640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf40b4420 0x7f0bf41c7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40422/0 (socket says 172.21.15.3:40422) 2023-10-30T12:32:34.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf93df640 1 -- 172.21.15.3:0/1829360479 learned_addr learned my addr 172.21.15.3:0/1829360479 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:34.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf93df640 10 monclient(hunting): get_auth_request con 0x7f0bf40b4420 auth_method 0 2023-10-30T12:32:34.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf93df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:34.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf93df640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:34.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf93df640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:34.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf9be0640 10 monclient(hunting): get_auth_request con 0x7f0bf40b12d0 auth_method 0 2023-10-30T12:32:34.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf9be0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:34.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf9be0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:34.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf9be0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:34.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bfa3e1640 10 monclient(hunting): get_auth_request con 0x7f0bf4141910 auth_method 0 2023-10-30T12:32:34.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bfa3e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:34.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bfa3e1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:34.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bfa3e1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:34.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf93df640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:34.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf93df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:34.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf93df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:34.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf9be0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:34.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf9be0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:34.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bf9be0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:34.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bfa3e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:34.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bfa3e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:34.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.577+0000 7f0bfa3e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:34.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.580+0000 7f0bfa3e1640 10 monclient(hunting): handle_auth_done global_id 21832 payload 824 2023-10-30T12:32:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.580+0000 7f0bfa3e1640 1 -- 172.21.15.3:0/1829360479 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf40b12d0 msgr2=0x7f0bf41cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.580+0000 7f0bfa3e1640 1 --2- 172.21.15.3:0/1829360479 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf40b12d0 0x7f0bf41cb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.580+0000 7f0bfa3e1640 1 -- 172.21.15.3:0/1829360479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf40b4420 msgr2=0x7f0bf41c7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.580+0000 7f0bfa3e1640 1 --2- 172.21.15.3:0/1829360479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf40b4420 0x7f0bf41c7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.580+0000 7f0bfa3e1640 10 monclient: _finish_hunting 0 2023-10-30T12:32:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.580+0000 7f0bfa3e1640 1 monclient: found mon.noname-b 2023-10-30T12:32:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.580+0000 7f0bfa3e1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.580+0000 7f0bfa3e1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.580+0000 7f0bfa3e1640 1 -- 172.21.15.3:0/1829360479 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf41d6740 con 0x7f0bf4141910 2023-10-30T12:32:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.582+0000 7f0bfa3e1640 1 --2- 172.21.15.3:0/1829360479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bf4141910 0x7f0bf41d6eb0 secure :-1 s=READY pgs=5308 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40b4cf0 tx=0x7f0bec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.582+0000 7f0bea7fc640 10 client.?.objecter ms_handle_connect 0x7f0bf4141910 2023-10-30T12:32:34.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.582+0000 7f0bea7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:34.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bea7fc640 1 -- 172.21.15.3:0/1829360479 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec031f00 con 0x7f0bf4141910 2023-10-30T12:32:34.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:34.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:34.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bea7fc640 10 monclient: dump: 2023-10-30T12:32:34.613 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:34.613 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:34.613 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.614 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.614 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:34.614 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:34.614 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:34.614 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:34.615 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:34.615 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:34.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bea7fc640 10 monclient: _finish_auth 0 2023-10-30T12:32:34.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bea7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:32:34.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:34.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bea7fc640 1 -- 172.21.15.3:0/1829360479 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec03f030 con 0x7f0bf4141910 2023-10-30T12:32:34.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bea7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:34.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bfbe44640 5 monclient: authenticate success, global_id 21832 2023-10-30T12:32:34.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bea7fc640 1 -- 172.21.15.3:0/1829360479 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec032aa0 con 0x7f0bf4141910 2023-10-30T12:32:34.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:34.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:34.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bea7fc640 10 monclient: dump: 2023-10-30T12:32:34.617 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:34.617 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:34.617 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.617 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.618 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:34.618 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:34.618 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:34.618 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:34.618 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:34.619 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:34.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bfbe44640 10 monclient: _renew_subs 2023-10-30T12:32:34.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bebfff640 4 set_mon_vals no callback set 2023-10-30T12:32:34.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bfbe44640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:34.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bfbe44640 1 -- 172.21.15.3:0/1829360479 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf41cb2d0 con 0x7f0bf4141910 2023-10-30T12:32:34.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bfbe44640 10 client.21832.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:34.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bfbe44640 10 monclient: _renew_subs 2023-10-30T12:32:34.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bfbe44640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:34.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bfbe44640 1 -- 172.21.15.3:0/1829360479 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf41d75d0 con 0x7f0bf4141910 2023-10-30T12:32:34.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.586+0000 7f0bfbe44640 1 librados: init done 2023-10-30T12:32:34.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.587+0000 7f0bfbe44640 10 client.21832.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:34.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.587+0000 7f0bfbe44640 15 client.21832.objecter _session_command_op_assign -1 1 2023-10-30T12:32:34.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.587+0000 7f0bfbe44640 20 client.21832.objecter _get_session osd=-1 returning homeless 2023-10-30T12:32:34.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.587+0000 7f0bfbe44640 10 client.21832.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:34.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.587+0000 7f0bfbe44640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:34.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.587+0000 7f0bfbe44640 1 -- 172.21.15.3:0/1829360479 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0bf413f4d0 con 0x7f0bf4141910 2023-10-30T12:32:34.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.605+0000 7f0bea7fc640 1 -- 172.21.15.3:0/1829360479 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f0bec032c10 con 0x7f0bf4141910 2023-10-30T12:32:34.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.606+0000 7f0bea7fc640 1 --2- 172.21.15.3:0/1829360479 >> v2:172.21.15.92:6800/1750155735 conn(0x7f0bd806c810 0x7f0bd806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.606+0000 7f0bf9be0640 1 --2- 172.21.15.3:0/1829360479 >> v2:172.21.15.92:6800/1750155735 conn(0x7f0bd806c810 0x7f0bd806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.606+0000 7f0bf9be0640 10 monclient: get_auth_request con 0x7f0bd806c810 auth_method 0 2023-10-30T12:32:34.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.606+0000 7f0bf9be0640 1 --2- 172.21.15.3:0/1829360479 >> v2:172.21.15.92:6800/1750155735 conn(0x7f0bd806c810 0x7f0bd806ecd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00113e0 tx=0x7f0bf000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:34.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.606+0000 7f0bea7fc640 10 client.21832.objecter ms_handle_connect 0x7f0bd806c810 2023-10-30T12:32:34.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 1 -- 172.21.15.3:0/1829360479 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f0bec079fb0 con 0x7f0bf4141910 2023-10-30T12:32:34.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 10 client.21832.objecter ms_dispatch 0x7f0bf4000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:34.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 3 client.21832.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:34.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 3 client.21832.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:34.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 10 client.21832.objecter checking command 1 2023-10-30T12:32:34.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 1 --2- 172.21.15.3:0/1829360479 >> v2:172.21.15.3:6800/1407765917 conn(0x7f0bd8073a80 0x7f0bd8075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 20 client.21832.objecter _get_session s=0x7f0bec032aa0 osd=2 3 2023-10-30T12:32:34.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 20 client.21832.objecter put_session s=0x7f0bec032aa0 osd=2 3 2023-10-30T12:32:34.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 15 client.21832.objecter _session_command_op_remove -1 1 2023-10-30T12:32:34.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 20 client.21832.objecter _get_session s=0x7f0bec032aa0 osd=2 3 2023-10-30T12:32:34.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 20 client.21832.objecter get_session s=0x7f0bec032aa0 osd=2 3 2023-10-30T12:32:34.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 15 client.21832.objecter _session_command_op_assign 2 1 2023-10-30T12:32:34.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 20 client.21832.objecter put_session s=0x7f0bec032aa0 osd=2 4 2023-10-30T12:32:34.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 10 client.21832.objecter _send_command 1 2023-10-30T12:32:34.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 1 -- 172.21.15.3:0/1829360479 --> v2:172.21.15.3:6800/1407765917 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0bec031f00 con 0x7f0bd8073a80 2023-10-30T12:32:34.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 20 client.21832.objecter dump_active .. 0 homeless 2023-10-30T12:32:34.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 1 -- 172.21.15.3:0/1829360479 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bec0094d0 con 0x7f0bf4141910 2023-10-30T12:32:34.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.609+0000 7f0bea7fc640 10 monclient: handle_get_version_reply finishing 1 version 900 2023-10-30T12:32:34.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.610+0000 7f0bf93df640 1 --2- 172.21.15.3:0/1829360479 >> v2:172.21.15.3:6800/1407765917 conn(0x7f0bd8073a80 0x7f0bd8075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.610+0000 7f0bf93df640 10 monclient: get_auth_request con 0x7f0bd8073a80 auth_method 0 2023-10-30T12:32:34.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.613+0000 7f0bf93df640 1 --2- 172.21.15.3:0/1829360479 >> v2:172.21.15.3:6800/1407765917 conn(0x7f0bd8073a80 0x7f0bd8075f20 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:34.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.613+0000 7f0bea7fc640 10 client.21832.objecter ms_handle_connect 0x7f0bd8073a80 2023-10-30T12:32:34.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.636+0000 7f0bea7fc640 1 -- 172.21.15.3:0/1829360479 <== osd.2 v2:172.21.15.3:6800/1407765917 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f0bec031f00 con 0x7f0bd8073a80 2023-10-30T12:32:34.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.636+0000 7f0bea7fc640 10 client.21832.objecter ms_dispatch 0x7f0bf4000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:32:34.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.636+0000 7f0bea7fc640 10 client.21832.objecter _finish_command 1 = system:0 2023-10-30T12:32:34.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.636+0000 7f0bea7fc640 20 client.21832.objecter put_session s=0x7f0bec032aa0 osd=2 4 2023-10-30T12:32:34.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.636+0000 7f0bea7fc640 15 client.21832.objecter _session_command_op_remove 2 1 2023-10-30T12:32:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.649+0000 7f0bfbe44640 10 client.21832.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:32:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.649+0000 7f0bfbe44640 15 client.21832.objecter _session_command_op_assign -1 2 2023-10-30T12:32:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.649+0000 7f0bfbe44640 20 client.21832.objecter _get_session s=0x7f0bec032aa0 osd=2 3 2023-10-30T12:32:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.649+0000 7f0bfbe44640 20 client.21832.objecter put_session s=0x7f0bec032aa0 osd=2 3 2023-10-30T12:32:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.649+0000 7f0bfbe44640 20 client.21832.objecter _get_session s=0x7f0bec032aa0 osd=2 3 2023-10-30T12:32:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.649+0000 7f0bfbe44640 15 client.21832.objecter _session_command_op_remove -1 2 2023-10-30T12:32:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.649+0000 7f0bfbe44640 20 client.21832.objecter get_session s=0x7f0bec032aa0 osd=2 3 2023-10-30T12:32:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.649+0000 7f0bfbe44640 15 client.21832.objecter _session_command_op_assign 2 2 2023-10-30T12:32:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.649+0000 7f0bfbe44640 20 client.21832.objecter put_session s=0x7f0bec032aa0 osd=2 4 2023-10-30T12:32:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.649+0000 7f0bfbe44640 10 client.21832.objecter _send_command 2 2023-10-30T12:32:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.649+0000 7f0bfbe44640 1 -- 172.21.15.3:0/1829360479 --> v2:172.21.15.3:6800/1407765917 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0bf40aa740 con 0x7f0bd8073a80 2023-10-30T12:32:34.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.663+0000 7f0bea7fc640 1 -- 172.21.15.3:0/1829360479 <== osd.2 v2:172.21.15.3:6800/1407765917 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f0be4009930 con 0x7f0bd8073a80 2023-10-30T12:32:34.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.663+0000 7f0bea7fc640 10 client.21832.objecter ms_dispatch 0x7f0bf4000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:32:34.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.663+0000 7f0bea7fc640 10 client.21832.objecter _finish_command 2 = system:0 2023-10-30T12:32:34.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.663+0000 7f0bea7fc640 20 client.21832.objecter put_session s=0x7f0bec032aa0 osd=2 4 2023-10-30T12:32:34.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.663+0000 7f0bea7fc640 15 client.21832.objecter _session_command_op_remove 2 2 2023-10-30T12:32:34.666 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-10-30T12:32:34.666 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:32:34.667 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-10-30T12:32:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 10 librados: watch_flush enter 2023-10-30T12:32:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 10 librados: watch_flush exit 2023-10-30T12:32:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 10 client.21832.objecter close_session for osd.2 2023-10-30T12:32:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 1 -- 172.21.15.3:0/1829360479 >> v2:172.21.15.3:6800/1407765917 conn(0x7f0bd8073a80 msgr2=0x7f0bd8075f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 1 --2- 172.21.15.3:0/1829360479 >> v2:172.21.15.3:6800/1407765917 conn(0x7f0bd8073a80 0x7f0bd8075f20 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 20 client.21832.objecter put_session s=0x7f0bec032aa0 osd=2 1 2023-10-30T12:32:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 20 client.21832.objecter shutdown clearing up homeless session... 2023-10-30T12:32:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 10 client.21832.objecter successfully canceled tick 2023-10-30T12:32:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 1 -- 172.21.15.3:0/1829360479 >> v2:172.21.15.92:6800/1750155735 conn(0x7f0bd806c810 msgr2=0x7f0bd806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 1 --2- 172.21.15.3:0/1829360479 >> v2:172.21.15.92:6800/1750155735 conn(0x7f0bd806c810 0x7f0bd806ecd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00113e0 tx=0x7f0bf000b040 comp rx=0 tx=0).stop 2023-10-30T12:32:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 10 monclient: shutdown 2023-10-30T12:32:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 1 -- 172.21.15.3:0/1829360479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bf4141910 msgr2=0x7f0bf41d6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 1 --2- 172.21.15.3:0/1829360479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bf4141910 0x7f0bf41d6eb0 secure :-1 s=READY pgs=5308 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40b4cf0 tx=0x7f0bec0046c0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfa3e1640 1 -- 172.21.15.3:0/1829360479 reap_dead start 2023-10-30T12:32:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 1 -- 172.21.15.3:0/1829360479 shutdown_connections 2023-10-30T12:32:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 1 -- 172.21.15.3:0/1829360479 >> 172.21.15.3:0/1829360479 conn(0x7f0bf4065ea0 msgr2=0x7f0bf41cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 1 -- 172.21.15.3:0/1829360479 shutdown_connections 2023-10-30T12:32:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 1 -- 172.21.15.3:0/1829360479 wait complete. 2023-10-30T12:32:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.664+0000 7f0bfbe44640 1 librados: shutdown 2023-10-30T12:32:34.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-30T12:32:34.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.781+0000 7ff1b861c640 10 monclient: get_monmap_and_config 2023-10-30T12:32:34.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.781+0000 7ff1b861c640 10 monclient: build_initial_monmap 2023-10-30T12:32:34.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.781+0000 7ff1b861c640 10 monclient: monmap: 2023-10-30T12:32:34.784 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:34.784 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:34.784 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:34.782664+0000 2023-10-30T12:32:34.784 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:34.782664+0000 2023-10-30T12:32:34.784 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:34.785 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:34.785 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:34.785 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:34.785 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:34.785 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:34.785 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:34.786 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:34.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.782+0000 7ff1b861c640 1 Processor -- start 2023-10-30T12:32:34.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.782+0000 7ff1b861c640 1 -- start start 2023-10-30T12:32:34.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.782+0000 7ff1b861c640 10 monclient: init 2023-10-30T12:32:34.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b861c640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:34.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b861c640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-10-30T12:32:34.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b861c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1b00ae690 addr v1:172.21.15.92:6789/0 2023-10-30T12:32:34.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b861c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff1b00aea70 0x7ff1b0070860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b861c640 10 monclient(hunting): picked mon.noname-c con 0x7ff1b00aea70 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:34.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b861c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff1b013cb60 0x7ff1b014a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b861c640 10 monclient(hunting): picked mon.noname-a con 0x7ff1b013cb60 addr v2:172.21.15.3:3300/0 2023-10-30T12:32:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b861c640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1b00703d0 con 0x7ff1b00ae690 2023-10-30T12:32:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b861c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b861c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1b0067b90 con 0x7ff1b013cb60 2023-10-30T12:32:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b861c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b861c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1b00a8700 con 0x7ff1b00aea70 2023-10-30T12:32:34.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b861c640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:34.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b861c640 10 monclient(hunting): authenticate will time out at 2745.997321s 2023-10-30T12:32:34.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b6bb9640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff1b013cb60 0x7ff1b014a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.783+0000 7ff1b5bb7640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff1b00aea70 0x7ff1b0070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b6bb9640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff1b013cb60 0x7ff1b014a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40424/0 (socket says 172.21.15.3:40424) 2023-10-30T12:32:34.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b6bb9640 1 -- 172.21.15.3:0/1999290870 learned_addr learned my addr 172.21.15.3:0/1999290870 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:34.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b6bb9640 10 monclient(hunting): get_auth_request con 0x7ff1b013cb60 auth_method 0 2023-10-30T12:32:34.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b6bb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:34.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b6bb9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:34.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b6bb9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:34.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b5bb7640 10 monclient(hunting): get_auth_request con 0x7ff1b00aea70 auth_method 0 2023-10-30T12:32:34.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b5bb7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:34.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b5bb7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:34.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b5bb7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:34.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b6bb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:34.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b6bb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:34.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b6bb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:34.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b5bb7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:34.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b5bb7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:34.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.784+0000 7ff1b5bb7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:34.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.786+0000 7ff1b6bb9640 10 monclient(hunting): handle_auth_done global_id 16650 payload 274 2023-10-30T12:32:34.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.786+0000 7ff1b6bb9640 1 -- 172.21.15.3:0/1999290870 >> v2:172.21.15.92:3300/0 conn(0x7ff1b00aea70 msgr2=0x7ff1b0070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:34.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.786+0000 7ff1b6bb9640 1 --2- 172.21.15.3:0/1999290870 >> v2:172.21.15.92:3300/0 conn(0x7ff1b00aea70 0x7ff1b0070860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.786+0000 7ff1b6bb9640 1 -- 172.21.15.3:0/1999290870 >> v1:172.21.15.92:6789/0 conn(0x7ff1b00ae690 legacy=0x7ff1b0142c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:34.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.786+0000 7ff1b6bb9640 10 monclient: _finish_hunting 0 2023-10-30T12:32:34.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.786+0000 7ff1b6bb9640 1 monclient: found mon.noname-a 2023-10-30T12:32:34.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.786+0000 7ff1b6bb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:34.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.786+0000 7ff1b6bb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:32:34.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.786+0000 7ff1b6bb9640 1 -- 172.21.15.3:0/1999290870 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b0070ee0 con 0x7ff1b013cb60 2023-10-30T12:32:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.788+0000 7ff1b6bb9640 1 --2- 172.21.15.3:0/1999290870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1b013cb60 0x7ff1b014a010 secure :-1 s=READY pgs=3609 cs=0 l=1 rev1=1 crypto rx=0x7ff1b006cc90 tx=0x7ff1a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532cca74f68dc0bc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b53b6640 1 -- 172.21.15.3:0/1999290870 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8031870 con 0x7ff1b013cb60 2023-10-30T12:32:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b53b6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b53b6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:32:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b53b6640 10 monclient: dump: 2023-10-30T12:32:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:34.804 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b53b6640 10 monclient: _finish_auth 0 2023-10-30T12:32:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b53b6640 10 monclient: _check_auth_tickets 2023-10-30T12:32:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b53b6640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b53b6640 1 -- 172.21.15.3:0/1999290870 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a80319e0 con 0x7ff1b013cb60 2023-10-30T12:32:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b861c640 5 monclient: authenticate success, global_id 16650 2023-10-30T12:32:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b861c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:32:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b53b6640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b861c640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b53b6640 1 -- 172.21.15.3:0/1999290870 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8031d80 con 0x7ff1b013cb60 2023-10-30T12:32:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b861c640 4 set_mon_vals no callback set 2023-10-30T12:32:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b53b6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b53b6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:32:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.791+0000 7ff1b53b6640 10 monclient: dump: 2023-10-30T12:32:34.807 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:34.807 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:34.807 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.807 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.807 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:34.808 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:34.808 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:34.808 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:34.808 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:34.808 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:34.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.792+0000 7ff1b861c640 10 monclient: shutdown 2023-10-30T12:32:34.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.792+0000 7ff1b861c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:34.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.792+0000 7ff1b861c640 1 -- 172.21.15.3:0/1999290870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1b013cb60 msgr2=0x7ff1b014a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.792+0000 7ff1b861c640 1 --2- 172.21.15.3:0/1999290870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1b013cb60 0x7ff1b014a010 secure :-1 s=READY pgs=3609 cs=0 l=1 rev1=1 crypto rx=0x7ff1b006cc90 tx=0x7ff1a802b910 comp rx=0 tx=0).stop 2023-10-30T12:32:34.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.792+0000 7ff1b861c640 1 -- 172.21.15.3:0/1999290870 shutdown_connections 2023-10-30T12:32:34.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.792+0000 7ff1b861c640 1 --2- 172.21.15.3:0/1999290870 >> v2:172.21.15.92:3300/0 conn(0x7ff1b00aea70 0x7ff1b0070860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.792+0000 7ff1b861c640 1 --2- 172.21.15.3:0/1999290870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1b013cb60 0x7ff1b014a010 unknown :-1 s=CLOSED pgs=3609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.792+0000 7ff1b861c640 1 -- 172.21.15.3:0/1999290870 >> 172.21.15.3:0/1999290870 conn(0x7ff1b0069d50 msgr2=0x7ff1b006a130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:34.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.792+0000 7ff1b861c640 1 -- 172.21.15.3:0/1999290870 shutdown_connections 2023-10-30T12:32:34.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.792+0000 7ff1b861c640 1 -- 172.21.15.3:0/1999290870 wait complete. 2023-10-30T12:32:34.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.793+0000 7ff1b861c640 10 monclient: build_initial_monmap 2023-10-30T12:32:34.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.793+0000 7ff1b861c640 10 monclient: monmap: 2023-10-30T12:32:34.811 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:34.811 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:34.811 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:34.811 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:34.812 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:34.812 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:34.812 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:34.812 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:34.812 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:34.813 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:34.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.793+0000 7ff1b861c640 1 librados: starting msgr at 2023-10-30T12:32:34.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.793+0000 7ff1b861c640 1 librados: starting objecter 2023-10-30T12:32:34.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.793+0000 7ff1b861c640 1 Processor -- start 2023-10-30T12:32:34.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.793+0000 7ff1b861c640 1 -- start start 2023-10-30T12:32:34.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.793+0000 7ff1b861c640 1 librados: setting wanted keys 2023-10-30T12:32:34.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.793+0000 7ff1b861c640 1 librados: calling monclient init 2023-10-30T12:32:34.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.793+0000 7ff1b861c640 10 monclient: init 2023-10-30T12:32:34.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:34.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:32:34.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1b00ae690 0x7ff1b00aaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 10 monclient(hunting): picked mon.noname-b con 0x7ff1b00ae690 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1b013cb60 0x7ff1b01d4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 10 monclient(hunting): picked mon.noname-c con 0x7ff1b013cb60 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1b0138100 0x7ff1b01385b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 10 monclient(hunting): picked mon.noname-a con 0x7ff1b0138100 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b013d390 con 0x7ff1b0138100 2023-10-30T12:32:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b01d5410 con 0x7ff1b00ae690 2023-10-30T12:32:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b01d5030 con 0x7ff1b013cb60 2023-10-30T12:32:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b861c640 10 monclient(hunting): authenticate will time out at 2746.008418s 2023-10-30T12:32:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b6bb9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1b0138100 0x7ff1b01385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.794+0000 7ff1b5bb7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1b013cb60 0x7ff1b01d4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b63b8640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1b00ae690 0x7ff1b00aaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b6bb9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1b0138100 0x7ff1b01385b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40436/0 (socket says 172.21.15.3:40436) 2023-10-30T12:32:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b6bb9640 1 -- 172.21.15.3:0/3171248482 learned_addr learned my addr 172.21.15.3:0/3171248482 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b6bb9640 10 monclient(hunting): get_auth_request con 0x7ff1b0138100 auth_method 0 2023-10-30T12:32:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b6bb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b6bb9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b6bb9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b63b8640 10 monclient(hunting): get_auth_request con 0x7ff1b00ae690 auth_method 0 2023-10-30T12:32:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b63b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b63b8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b63b8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b5bb7640 10 monclient(hunting): get_auth_request con 0x7ff1b013cb60 auth_method 0 2023-10-30T12:32:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b5bb7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b5bb7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b5bb7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b63b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b63b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b63b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b6bb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b6bb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.795+0000 7ff1b6bb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.796+0000 7ff1b5bb7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.796+0000 7ff1b5bb7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.796+0000 7ff1b5bb7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.798+0000 7ff1b63b8640 10 monclient(hunting): handle_auth_done global_id 21838 payload 824 2023-10-30T12:32:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.798+0000 7ff1b63b8640 1 -- 172.21.15.3:0/3171248482 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1b013cb60 msgr2=0x7ff1b01d4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.798+0000 7ff1b63b8640 1 --2- 172.21.15.3:0/3171248482 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1b013cb60 0x7ff1b01d4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.798+0000 7ff1b63b8640 1 -- 172.21.15.3:0/3171248482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1b0138100 msgr2=0x7ff1b01385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.798+0000 7ff1b63b8640 1 --2- 172.21.15.3:0/3171248482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1b0138100 0x7ff1b01385b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.798+0000 7ff1b63b8640 10 monclient: _finish_hunting 0 2023-10-30T12:32:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.798+0000 7ff1b63b8640 1 monclient: found mon.noname-b 2023-10-30T12:32:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.798+0000 7ff1b63b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.798+0000 7ff1b63b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:34.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.798+0000 7ff1b63b8640 1 -- 172.21.15.3:0/3171248482 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b0138af0 con 0x7ff1b00ae690 2023-10-30T12:32:34.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.800+0000 7ff1b63b8640 1 --2- 172.21.15.3:0/3171248482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1b00ae690 0x7ff1b00aaaf0 secure :-1 s=READY pgs=5309 cs=0 l=1 rev1=1 crypto rx=0x7ff1a000c6d0 tx=0x7ff1a000cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:34.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.800+0000 7ff1aeffd640 10 client.?.objecter ms_handle_connect 0x7ff1b00ae690 2023-10-30T12:32:34.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.800+0000 7ff1aeffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:34.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1aeffd640 1 -- 172.21.15.3:0/3171248482 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0003b70 con 0x7ff1b00ae690 2023-10-30T12:32:34.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:34.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:34.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1aeffd640 10 monclient: dump: 2023-10-30T12:32:34.827 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:34.827 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:34.828 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.828 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.828 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:34.828 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:34.828 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:34.828 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:34.829 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:34.829 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:34.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1aeffd640 10 monclient: _finish_auth 0 2023-10-30T12:32:34.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1aeffd640 10 monclient: _check_auth_tickets 2023-10-30T12:32:34.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:34.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1aeffd640 1 -- 172.21.15.3:0/3171248482 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a000e040 con 0x7ff1b00ae690 2023-10-30T12:32:34.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1b861c640 5 monclient: authenticate success, global_id 21838 2023-10-30T12:32:34.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1b861c640 10 monclient: _renew_subs 2023-10-30T12:32:34.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1b861c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:34.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1b861c640 1 -- 172.21.15.3:0/3171248482 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b00ab740 con 0x7ff1b00ae690 2023-10-30T12:32:34.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1aeffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:34.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1aeffd640 1 -- 172.21.15.3:0/3171248482 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0005070 con 0x7ff1b00ae690 2023-10-30T12:32:34.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:34.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:34.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1aeffd640 10 monclient: dump: 2023-10-30T12:32:34.832 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:34.832 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:34.832 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.832 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:34.832 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:34.833 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:34.833 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:34.833 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:34.833 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:34.833 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:34.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1b4bb5640 4 set_mon_vals no callback set 2023-10-30T12:32:34.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1b861c640 10 client.21838.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:34.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1b861c640 10 monclient: _renew_subs 2023-10-30T12:32:34.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1b861c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:34.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1b861c640 1 -- 172.21.15.3:0/3171248482 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b013d7c0 con 0x7ff1b00ae690 2023-10-30T12:32:34.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.804+0000 7ff1b861c640 1 librados: init done 2023-10-30T12:32:34.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.805+0000 7ff1b861c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:34.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.805+0000 7ff1b861c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:34.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.805+0000 7ff1b861c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:34.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.805+0000 7ff1b861c640 1 -- 172.21.15.3:0/3171248482 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b011b760 con 0x7ff1b00ae690 2023-10-30T12:32:34.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.822+0000 7ff1aeffd640 1 -- 172.21.15.3:0/3171248482 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7ff1a00053f0 con 0x7ff1b00ae690 2023-10-30T12:32:34.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.823+0000 7ff1aeffd640 1 --2- 172.21.15.3:0/3171248482 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff19c06c810 0x7ff19c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.823+0000 7ff1b5bb7640 1 --2- 172.21.15.3:0/3171248482 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff19c06c810 0x7ff19c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.823+0000 7ff1b5bb7640 10 monclient: get_auth_request con 0x7ff19c06c810 auth_method 0 2023-10-30T12:32:34.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.823+0000 7ff1b5bb7640 1 --2- 172.21.15.3:0/3171248482 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff19c06c810 0x7ff19c06ecd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00ab2f0 tx=0x7ff1a4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:34.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.823+0000 7ff1aeffd640 10 client.21838.objecter ms_handle_connect 0x7ff19c06c810 2023-10-30T12:32:34.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.826+0000 7ff1aeffd640 1 -- 172.21.15.3:0/3171248482 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7ff1a0057ce0 con 0x7ff1b00ae690 2023-10-30T12:32:34.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.826+0000 7ff1aeffd640 10 client.21838.objecter ms_dispatch 0x7ff1b0000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:34.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.826+0000 7ff1aeffd640 3 client.21838.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:34.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.826+0000 7ff1aeffd640 3 client.21838.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:34.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:34.826+0000 7ff1aeffd640 20 client.21838.objecter dump_active .. 0 homeless 2023-10-30T12:32:35.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.057+0000 7ff1aeffd640 1 -- 172.21.15.3:0/3171248482 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff1a005bd80 con 0x7ff1b00ae690 2023-10-30T12:32:35.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.057+0000 7ff1aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:35.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.057+0000 7ff1aeffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:32:35.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.237+0000 7ff1b861c640 1 -- 172.21.15.3:0/3171248482 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1b01d5200 con 0x7ff19c06c810 2023-10-30T12:32:35.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.238+0000 7ff1aeffd640 1 -- 172.21.15.3:0/3171248482 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff1a4011a30 con 0x7ff19c06c810 2023-10-30T12:32:35.240 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 2 to deep-scrub 2023-10-30T12:32:35.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 10 librados: watch_flush enter 2023-10-30T12:32:35.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 10 librados: watch_flush exit 2023-10-30T12:32:35.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 20 client.21838.objecter shutdown clearing up homeless session... 2023-10-30T12:32:35.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 10 client.21838.objecter successfully canceled tick 2023-10-30T12:32:35.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 1 -- 172.21.15.3:0/3171248482 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff19c06c810 msgr2=0x7ff19c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:35.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 1 --2- 172.21.15.3:0/3171248482 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff19c06c810 0x7ff19c06ecd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00ab2f0 tx=0x7ff1a4009190 comp rx=0 tx=0).stop 2023-10-30T12:32:35.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 10 monclient: shutdown 2023-10-30T12:32:35.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:35.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 1 -- 172.21.15.3:0/3171248482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1b00ae690 msgr2=0x7ff1b00aaaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:35.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 1 --2- 172.21.15.3:0/3171248482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1b00ae690 0x7ff1b00aaaf0 secure :-1 s=READY pgs=5309 cs=0 l=1 rev1=1 crypto rx=0x7ff1a000c6d0 tx=0x7ff1a000cb90 comp rx=0 tx=0).stop 2023-10-30T12:32:35.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 1 -- 172.21.15.3:0/3171248482 shutdown_connections 2023-10-30T12:32:35.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 1 --2- 172.21.15.3:0/3171248482 >> v2:172.21.15.92:6800/1750155735 conn(0x7ff19c06c810 0x7ff19c06ecd0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 1 --2- 172.21.15.3:0/3171248482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1b0138100 0x7ff1b01385b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 1 --2- 172.21.15.3:0/3171248482 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1b013cb60 0x7ff1b01d4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 1 --2- 172.21.15.3:0/3171248482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1b00ae690 0x7ff1b00aaaf0 unknown :-1 s=CLOSED pgs=5309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 1 -- 172.21.15.3:0/3171248482 >> 172.21.15.3:0/3171248482 conn(0x7ff1b0069d50 msgr2=0x7ff1b013c340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:35.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 1 -- 172.21.15.3:0/3171248482 shutdown_connections 2023-10-30T12:32:35.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 1 -- 172.21.15.3:0/3171248482 wait complete. 2023-10-30T12:32:35.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.240+0000 7ff1b861c640 1 librados: shutdown 2023-10-30T12:32:35.252 INFO:tasks.ceph:Scrubbing osd.3 2023-10-30T12:32:35.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:32:35.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.363+0000 7f5e657e9640 10 monclient: get_monmap_and_config 2023-10-30T12:32:35.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.363+0000 7f5e657e9640 10 monclient: build_initial_monmap 2023-10-30T12:32:35.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.363+0000 7f5e657e9640 10 monclient: monmap: 2023-10-30T12:32:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:35.364471+0000 2023-10-30T12:32:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:35.364471+0000 2023-10-30T12:32:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:35.368 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:35.368 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:35.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 1 Processor -- start 2023-10-30T12:32:35.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 1 -- start start 2023-10-30T12:32:35.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 10 monclient: init 2023-10-30T12:32:35.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:35.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-10-30T12:32:35.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e600b4360 addr v1:172.21.15.3:6789/0 2023-10-30T12:32:35.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e6013f320 0x7f5e60144e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:35.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 10 monclient(hunting): picked mon.noname-b con 0x7f5e6013f320 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:35.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5e600b12b0 0x7f5e60145350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:35.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 10 monclient(hunting): picked mon.noname-c con 0x7f5e600b12b0 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:35.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e600aa180 con 0x7f5e600b4360 2023-10-30T12:32:35.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:35.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e60067a30 con 0x7f5e600b12b0 2023-10-30T12:32:35.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:35.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e600b4740 con 0x7f5e6013f320 2023-10-30T12:32:35.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:35.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.364+0000 7f5e657e9640 10 monclient(hunting): authenticate will time out at 2746.578875s 2023-10-30T12:32:35.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.398+0000 7f5e5f7fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5e600b12b0 0x7f5e60145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:35.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.399+0000 7f5e5e7fc640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e6013f320 0x7f5e60144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:35.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.399+0000 7f5e5f7fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5e600b12b0 0x7f5e60145350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44654/0 (socket says 172.21.15.3:44654) 2023-10-30T12:32:35.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.399+0000 7f5e5f7fe640 1 -- 172.21.15.3:0/490296794 learned_addr learned my addr 172.21.15.3:0/490296794 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:35.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.399+0000 7f5e5f7fe640 10 monclient(hunting): get_auth_request con 0x7f5e600b12b0 auth_method 0 2023-10-30T12:32:35.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.399+0000 7f5e5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:35.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.399+0000 7f5e5f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:35.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.399+0000 7f5e5f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:35.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.399+0000 7f5e5e7fc640 10 monclient(hunting): get_auth_request con 0x7f5e6013f320 auth_method 0 2023-10-30T12:32:35.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.399+0000 7f5e5e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:35.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.399+0000 7f5e5e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:35.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.399+0000 7f5e5e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:35.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.400+0000 7f5e5e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:35.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.400+0000 7f5e5e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:35.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.400+0000 7f5e5e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:35.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.401+0000 7f5e5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:35.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.401+0000 7f5e5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:35.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.401+0000 7f5e5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:35.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.402+0000 7f5e5e7fc640 10 monclient(hunting): handle_auth_done global_id 15575 payload 274 2023-10-30T12:32:35.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.402+0000 7f5e5e7fc640 1 -- 172.21.15.3:0/490296794 >> v2:172.21.15.92:3300/0 conn(0x7f5e600b12b0 msgr2=0x7f5e60145350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:35.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.402+0000 7f5e5e7fc640 1 --2- 172.21.15.3:0/490296794 >> v2:172.21.15.92:3300/0 conn(0x7f5e600b12b0 0x7f5e60145350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.402+0000 7f5e5e7fc640 1 -- 172.21.15.3:0/490296794 >> v1:172.21.15.3:6789/0 conn(0x7f5e600b4360 legacy=0x7f5e6013ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:35.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.402+0000 7f5e5e7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:32:35.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.402+0000 7f5e5e7fc640 1 monclient: found mon.noname-b 2023-10-30T12:32:35.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.402+0000 7f5e5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:35.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.402+0000 7f5e5e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:32:35.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.402+0000 7f5e5e7fc640 1 -- 172.21.15.3:0/490296794 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e600aa3d0 con 0x7f5e6013f320 2023-10-30T12:32:35.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.406+0000 7f5e5e7fc640 1 --2- 172.21.15.3:0/490296794 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e6013f320 0x7f5e60144e10 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f5e540097a0 tx=0x7f5e5402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea3512dc5c020b9d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:35.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e5dffb640 1 -- 172.21.15.3:0/490296794 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54031700 con 0x7f5e6013f320 2023-10-30T12:32:35.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:35.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e5dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:35.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e5dffb640 10 monclient: dump: 2023-10-30T12:32:35.413 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:35.413 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:35.413 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.413 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.414 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:35.414 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:35.414 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:35.414 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:35.415 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:35.415 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:35.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e5dffb640 10 monclient: _finish_auth 0 2023-10-30T12:32:35.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e5dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:32:35.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:35.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e5dffb640 1 -- 172.21.15.3:0/490296794 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e54031870 con 0x7f5e6013f320 2023-10-30T12:32:35.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e657e9640 5 monclient: authenticate success, global_id 15575 2023-10-30T12:32:35.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e657e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:32:35.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e5dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:35.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e657e9640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:35.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e657e9640 4 set_mon_vals no callback set 2023-10-30T12:32:35.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e5dffb640 1 -- 172.21.15.3:0/490296794 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54031c10 con 0x7f5e6013f320 2023-10-30T12:32:35.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:35.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e5dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:32:35.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.410+0000 7f5e5dffb640 10 monclient: dump: 2023-10-30T12:32:35.418 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:35.418 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:35.418 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.418 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.419 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:35.419 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:35.419 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:35.419 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:35.420 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:35.420 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:35.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.411+0000 7f5e657e9640 10 monclient: shutdown 2023-10-30T12:32:35.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.411+0000 7f5e657e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:35.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.411+0000 7f5e657e9640 1 -- 172.21.15.3:0/490296794 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e6013f320 msgr2=0x7f5e60144e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:35.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.411+0000 7f5e657e9640 1 --2- 172.21.15.3:0/490296794 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e6013f320 0x7f5e60144e10 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f5e540097a0 tx=0x7f5e5402b7a0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.411+0000 7f5e657e9640 1 -- 172.21.15.3:0/490296794 shutdown_connections 2023-10-30T12:32:35.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.411+0000 7f5e657e9640 1 --2- 172.21.15.3:0/490296794 >> v2:172.21.15.92:3300/0 conn(0x7f5e600b12b0 0x7f5e60145350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.411+0000 7f5e657e9640 1 --2- 172.21.15.3:0/490296794 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e6013f320 0x7f5e60144e10 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.411+0000 7f5e657e9640 1 -- 172.21.15.3:0/490296794 >> 172.21.15.3:0/490296794 conn(0x7f5e60065ed0 msgr2=0x7f5e600662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:35.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.411+0000 7f5e657e9640 1 -- 172.21.15.3:0/490296794 shutdown_connections 2023-10-30T12:32:35.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.411+0000 7f5e657e9640 1 -- 172.21.15.3:0/490296794 wait complete. 2023-10-30T12:32:35.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.412+0000 7f5e657e9640 10 monclient: build_initial_monmap 2023-10-30T12:32:35.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.412+0000 7f5e657e9640 10 monclient: monmap: 2023-10-30T12:32:35.428 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:35.428 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:35.428 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:35.428 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:35.429 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:35.429 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:35.429 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:35.429 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:35.429 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:35.430 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:35.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.412+0000 7f5e657e9640 1 librados: starting msgr at 2023-10-30T12:32:35.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.412+0000 7f5e657e9640 1 librados: starting objecter 2023-10-30T12:32:35.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.412+0000 7f5e657e9640 1 Processor -- start 2023-10-30T12:32:35.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.412+0000 7f5e657e9640 1 -- start start 2023-10-30T12:32:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.412+0000 7f5e657e9640 1 librados: setting wanted keys 2023-10-30T12:32:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.412+0000 7f5e657e9640 1 librados: calling monclient init 2023-10-30T12:32:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.412+0000 7f5e657e9640 10 monclient: init 2023-10-30T12:32:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:32:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e600b12b0 0x7f5e601408a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 10 monclient(hunting): picked mon.noname-c con 0x7f5e600b12b0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e600b4360 0x7f5e601d9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 10 monclient(hunting): picked mon.noname-b con 0x7f5e600b4360 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e6013fb00 0x7f5e601da330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 10 monclient(hunting): picked mon.noname-a con 0x7f5e6013fb00 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e60145ad0 con 0x7f5e6013fb00 2023-10-30T12:32:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e60142c10 con 0x7f5e600b4360 2023-10-30T12:32:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e601da870 con 0x7f5e600b12b0 2023-10-30T12:32:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e657e9640 10 monclient(hunting): authenticate will time out at 2746.627530s 2023-10-30T12:32:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e5effd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e600b12b0 0x7f5e601408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.413+0000 7f5e5f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e6013fb00 0x7f5e601da330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5e7fc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e600b4360 0x7f5e601d9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5effd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e600b12b0 0x7f5e601408a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:37644/0 (socket says 172.21.15.3:37644) 2023-10-30T12:32:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5effd640 1 -- 172.21.15.3:0/547221787 learned_addr learned my addr 172.21.15.3:0/547221787 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5effd640 10 monclient(hunting): get_auth_request con 0x7f5e600b12b0 auth_method 0 2023-10-30T12:32:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:35.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5f7fe640 10 monclient(hunting): get_auth_request con 0x7f5e6013fb00 auth_method 0 2023-10-30T12:32:35.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:35.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:35.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:35.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5e7fc640 10 monclient(hunting): get_auth_request con 0x7f5e600b4360 auth_method 0 2023-10-30T12:32:35.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:35.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:35.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:35.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:35.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:35.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:35.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:35.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:35.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:35.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:35.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:35.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.414+0000 7f5e5e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:35.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.417+0000 7f5e5e7fc640 10 monclient(hunting): handle_auth_done global_id 21844 payload 824 2023-10-30T12:32:35.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.417+0000 7f5e5e7fc640 1 -- 172.21.15.3:0/547221787 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e600b12b0 msgr2=0x7f5e601408a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:35.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.417+0000 7f5e5e7fc640 1 --2- 172.21.15.3:0/547221787 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e600b12b0 0x7f5e601408a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.417+0000 7f5e5e7fc640 1 -- 172.21.15.3:0/547221787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e6013fb00 msgr2=0x7f5e601da330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:35.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.417+0000 7f5e5e7fc640 1 --2- 172.21.15.3:0/547221787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e6013fb00 0x7f5e601da330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.417+0000 7f5e5e7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:32:35.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.417+0000 7f5e5e7fc640 1 monclient: found mon.noname-b 2023-10-30T12:32:35.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.417+0000 7f5e5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:35.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.417+0000 7f5e5e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:35.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.417+0000 7f5e5e7fc640 1 -- 172.21.15.3:0/547221787 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e60144810 con 0x7f5e600b4360 2023-10-30T12:32:35.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.419+0000 7f5e5e7fc640 1 --2- 172.21.15.3:0/547221787 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e600b4360 0x7f5e601d9df0 secure :-1 s=READY pgs=5310 cs=0 l=1 rev1=1 crypto rx=0x7f5e54033000 tx=0x7f5e540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:35.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.419+0000 7f5e477fe640 10 client.?.objecter ms_handle_connect 0x7f5e600b4360 2023-10-30T12:32:35.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.419+0000 7f5e477fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:35.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e477fe640 1 -- 172.21.15.3:0/547221787 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54033c30 con 0x7f5e600b4360 2023-10-30T12:32:35.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:35.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:35.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e477fe640 10 monclient: dump: 2023-10-30T12:32:35.444 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:35.444 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:35.445 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.445 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.445 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:35.445 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:35.445 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:35.446 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:35.446 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:35.446 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:35.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e477fe640 10 monclient: _finish_auth 0 2023-10-30T12:32:35.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e477fe640 10 monclient: _check_auth_tickets 2023-10-30T12:32:35.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:35.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e477fe640 1 -- 172.21.15.3:0/547221787 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e54033de0 con 0x7f5e600b4360 2023-10-30T12:32:35.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e477fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:35.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e477fe640 1 -- 172.21.15.3:0/547221787 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5403d6b0 con 0x7f5e600b4360 2023-10-30T12:32:35.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e657e9640 5 monclient: authenticate success, global_id 21844 2023-10-30T12:32:35.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e657e9640 10 monclient: _renew_subs 2023-10-30T12:32:35.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e657e9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:35.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e657e9640 1 -- 172.21.15.3:0/547221787 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e60140de0 con 0x7f5e600b4360 2023-10-30T12:32:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e5d7fa640 4 set_mon_vals no callback set 2023-10-30T12:32:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e477fe640 10 monclient: dump: 2023-10-30T12:32:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:35.451 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e657e9640 10 client.21844.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e657e9640 10 monclient: _renew_subs 2023-10-30T12:32:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e657e9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e657e9640 1 -- 172.21.15.3:0/547221787 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e601d66f0 con 0x7f5e600b4360 2023-10-30T12:32:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.423+0000 7f5e657e9640 1 librados: init done 2023-10-30T12:32:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.424+0000 7f5e657e9640 10 client.21844.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.424+0000 7f5e657e9640 15 client.21844.objecter _session_command_op_assign -1 1 2023-10-30T12:32:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.424+0000 7f5e657e9640 20 client.21844.objecter _get_session osd=-1 returning homeless 2023-10-30T12:32:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.424+0000 7f5e657e9640 10 client.21844.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.424+0000 7f5e657e9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:35.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.424+0000 7f5e657e9640 1 -- 172.21.15.3:0/547221787 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5e600a90e0 con 0x7f5e600b4360 2023-10-30T12:32:35.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.443+0000 7f5e477fe640 1 -- 172.21.15.3:0/547221787 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f5e5403a040 con 0x7f5e600b4360 2023-10-30T12:32:35.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.443+0000 7f5e477fe640 1 --2- 172.21.15.3:0/547221787 >> v2:172.21.15.92:6800/1750155735 conn(0x7f5e4c06c810 0x7f5e4c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:35.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.443+0000 7f5e5effd640 1 --2- 172.21.15.3:0/547221787 >> v2:172.21.15.92:6800/1750155735 conn(0x7f5e4c06c810 0x7f5e4c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:35.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.444+0000 7f5e5effd640 10 monclient: get_auth_request con 0x7f5e4c06c810 auth_method 0 2023-10-30T12:32:35.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.444+0000 7f5e5effd640 1 --2- 172.21.15.3:0/547221787 >> v2:172.21.15.92:6800/1750155735 conn(0x7f5e4c06c810 0x7f5e4c06ecd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5e50005020 tx=0x7f5e50007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:35.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.444+0000 7f5e477fe640 10 client.21844.objecter ms_handle_connect 0x7f5e4c06c810 2023-10-30T12:32:35.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.445+0000 7f5e477fe640 1 -- 172.21.15.3:0/547221787 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f5e54079af0 con 0x7f5e600b4360 2023-10-30T12:32:35.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.445+0000 7f5e477fe640 10 client.21844.objecter ms_dispatch 0x7f5e60000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:35.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.445+0000 7f5e477fe640 3 client.21844.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:35.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.445+0000 7f5e477fe640 3 client.21844.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:35.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e477fe640 10 client.21844.objecter checking command 1 2023-10-30T12:32:35.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e477fe640 1 --2- 172.21.15.3:0/547221787 >> v2:172.21.15.3:6812/2654726311 conn(0x7f5e4c073a80 0x7f5e4c075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:35.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e477fe640 20 client.21844.objecter _get_session s=0x7f5e5403d6b0 osd=3 3 2023-10-30T12:32:35.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e477fe640 20 client.21844.objecter put_session s=0x7f5e5403d6b0 osd=3 3 2023-10-30T12:32:35.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e477fe640 15 client.21844.objecter _session_command_op_remove -1 1 2023-10-30T12:32:35.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e477fe640 20 client.21844.objecter _get_session s=0x7f5e5403d6b0 osd=3 3 2023-10-30T12:32:35.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e477fe640 20 client.21844.objecter get_session s=0x7f5e5403d6b0 osd=3 3 2023-10-30T12:32:35.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e477fe640 15 client.21844.objecter _session_command_op_assign 3 1 2023-10-30T12:32:35.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e477fe640 20 client.21844.objecter put_session s=0x7f5e5403d6b0 osd=3 4 2023-10-30T12:32:35.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e477fe640 10 client.21844.objecter _send_command 1 2023-10-30T12:32:35.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e477fe640 1 -- 172.21.15.3:0/547221787 --> v2:172.21.15.3:6812/2654726311 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5e54033c30 con 0x7f5e4c073a80 2023-10-30T12:32:35.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e477fe640 20 client.21844.objecter dump_active .. 0 homeless 2023-10-30T12:32:35.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e5f7fe640 1 --2- 172.21.15.3:0/547221787 >> v2:172.21.15.3:6812/2654726311 conn(0x7f5e4c073a80 0x7f5e4c075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:35.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e477fe640 1 -- 172.21.15.3:0/547221787 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e5402f050 con 0x7f5e600b4360 2023-10-30T12:32:35.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e477fe640 10 monclient: handle_get_version_reply finishing 1 version 900 2023-10-30T12:32:35.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.446+0000 7f5e5f7fe640 10 monclient: get_auth_request con 0x7f5e4c073a80 auth_method 0 2023-10-30T12:32:35.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.449+0000 7f5e5f7fe640 1 --2- 172.21.15.3:0/547221787 >> v2:172.21.15.3:6812/2654726311 conn(0x7f5e4c073a80 0x7f5e4c075f20 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:35.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.449+0000 7f5e477fe640 10 client.21844.objecter ms_handle_connect 0x7f5e4c073a80 2023-10-30T12:32:35.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.473+0000 7f5e477fe640 1 -- 172.21.15.3:0/547221787 <== osd.3 v2:172.21.15.3:6812/2654726311 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f5e54033c30 con 0x7f5e4c073a80 2023-10-30T12:32:35.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.473+0000 7f5e477fe640 10 client.21844.objecter ms_dispatch 0x7f5e60000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:32:35.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.473+0000 7f5e477fe640 10 client.21844.objecter _finish_command 1 = system:0 2023-10-30T12:32:35.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.473+0000 7f5e477fe640 20 client.21844.objecter put_session s=0x7f5e5403d6b0 osd=3 4 2023-10-30T12:32:35.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.473+0000 7f5e477fe640 15 client.21844.objecter _session_command_op_remove 3 1 2023-10-30T12:32:35.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.486+0000 7f5e657e9640 10 client.21844.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:32:35.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.486+0000 7f5e657e9640 15 client.21844.objecter _session_command_op_assign -1 2 2023-10-30T12:32:35.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.486+0000 7f5e657e9640 20 client.21844.objecter _get_session s=0x7f5e5403d6b0 osd=3 3 2023-10-30T12:32:35.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.486+0000 7f5e657e9640 20 client.21844.objecter put_session s=0x7f5e5403d6b0 osd=3 3 2023-10-30T12:32:35.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.486+0000 7f5e657e9640 20 client.21844.objecter _get_session s=0x7f5e5403d6b0 osd=3 3 2023-10-30T12:32:35.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.486+0000 7f5e657e9640 15 client.21844.objecter _session_command_op_remove -1 2 2023-10-30T12:32:35.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.486+0000 7f5e657e9640 20 client.21844.objecter get_session s=0x7f5e5403d6b0 osd=3 3 2023-10-30T12:32:35.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.486+0000 7f5e657e9640 15 client.21844.objecter _session_command_op_assign 3 2 2023-10-30T12:32:35.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.486+0000 7f5e657e9640 20 client.21844.objecter put_session s=0x7f5e5403d6b0 osd=3 4 2023-10-30T12:32:35.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.486+0000 7f5e657e9640 10 client.21844.objecter _send_command 2 2023-10-30T12:32:35.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.486+0000 7f5e657e9640 1 -- 172.21.15.3:0/547221787 --> v2:172.21.15.3:6812/2654726311 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5e600aa6e0 con 0x7f5e4c073a80 2023-10-30T12:32:35.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e477fe640 1 -- 172.21.15.3:0/547221787 <== osd.3 v2:172.21.15.3:6812/2654726311 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f5e5800f040 con 0x7f5e4c073a80 2023-10-30T12:32:35.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e477fe640 10 client.21844.objecter ms_dispatch 0x7f5e60000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:32:35.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e477fe640 10 client.21844.objecter _finish_command 2 = system:0 2023-10-30T12:32:35.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e477fe640 20 client.21844.objecter put_session s=0x7f5e5403d6b0 osd=3 4 2023-10-30T12:32:35.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e477fe640 15 client.21844.objecter _session_command_op_remove 3 2 2023-10-30T12:32:35.503 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-10-30T12:32:35.503 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:32:35.503 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-10-30T12:32:35.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e657e9640 10 librados: watch_flush enter 2023-10-30T12:32:35.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e657e9640 10 librados: watch_flush exit 2023-10-30T12:32:35.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e657e9640 10 client.21844.objecter close_session for osd.3 2023-10-30T12:32:35.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e657e9640 1 -- 172.21.15.3:0/547221787 >> v2:172.21.15.3:6812/2654726311 conn(0x7f5e4c073a80 msgr2=0x7f5e4c075f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:35.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e657e9640 1 --2- 172.21.15.3:0/547221787 >> v2:172.21.15.3:6812/2654726311 conn(0x7f5e4c073a80 0x7f5e4c075f20 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e657e9640 20 client.21844.objecter put_session s=0x7f5e5403d6b0 osd=3 1 2023-10-30T12:32:35.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e657e9640 20 client.21844.objecter shutdown clearing up homeless session... 2023-10-30T12:32:35.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e657e9640 10 client.21844.objecter successfully canceled tick 2023-10-30T12:32:35.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e657e9640 1 -- 172.21.15.3:0/547221787 >> v2:172.21.15.92:6800/1750155735 conn(0x7f5e4c06c810 msgr2=0x7f5e4c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:35.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e657e9640 1 --2- 172.21.15.3:0/547221787 >> v2:172.21.15.92:6800/1750155735 conn(0x7f5e4c06c810 0x7f5e4c06ecd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5e50005020 tx=0x7f5e50007ef0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e657e9640 10 monclient: shutdown 2023-10-30T12:32:35.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e657e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:35.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e657e9640 1 -- 172.21.15.3:0/547221787 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e600b4360 msgr2=0x7f5e601d9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:35.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.500+0000 7f5e657e9640 1 --2- 172.21.15.3:0/547221787 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e600b4360 0x7f5e601d9df0 secure :-1 s=READY pgs=5310 cs=0 l=1 rev1=1 crypto rx=0x7f5e54033000 tx=0x7f5e540046c0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.501+0000 7f5e5f7fe640 1 -- 172.21.15.3:0/547221787 reap_dead start 2023-10-30T12:32:35.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.501+0000 7f5e657e9640 1 -- 172.21.15.3:0/547221787 shutdown_connections 2023-10-30T12:32:35.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.501+0000 7f5e657e9640 1 -- 172.21.15.3:0/547221787 >> 172.21.15.3:0/547221787 conn(0x7f5e60065ed0 msgr2=0x7f5e600b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:35.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.501+0000 7f5e657e9640 1 -- 172.21.15.3:0/547221787 shutdown_connections 2023-10-30T12:32:35.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.501+0000 7f5e657e9640 1 -- 172.21.15.3:0/547221787 wait complete. 2023-10-30T12:32:35.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.501+0000 7f5e657e9640 1 librados: shutdown 2023-10-30T12:32:35.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-30T12:32:35.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.621+0000 7f2b27ca9640 10 monclient: get_monmap_and_config 2023-10-30T12:32:35.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.621+0000 7f2b27ca9640 10 monclient: build_initial_monmap 2023-10-30T12:32:35.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.621+0000 7f2b27ca9640 10 monclient: monmap: 2023-10-30T12:32:35.624 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:35.624 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:35.624 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:35.622249+0000 2023-10-30T12:32:35.624 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:35.622249+0000 2023-10-30T12:32:35.624 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:35.625 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:35.625 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:35.625 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:35.625 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:35.626 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:35.626 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:35.626 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:35.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.621+0000 7f2b27ca9640 1 Processor -- start 2023-10-30T12:32:35.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.622+0000 7f2b27ca9640 1 -- start start 2023-10-30T12:32:35.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.622+0000 7f2b27ca9640 10 monclient: init 2023-10-30T12:32:35.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.622+0000 7f2b27ca9640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:35.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.622+0000 7f2b27ca9640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-10-30T12:32:35.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.622+0000 7f2b27ca9640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2b200a8e40 0x7f2b2013efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:35.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.622+0000 7f2b27ca9640 10 monclient(hunting): picked mon.noname-c con 0x7f2b200a8e40 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:35.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.622+0000 7f2b27ca9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b200b13c0 0x7f2b20141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:35.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.622+0000 7f2b27ca9640 10 monclient(hunting): picked mon.noname-b con 0x7f2b200b13c0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:35.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.622+0000 7f2b27ca9640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2b2013f500 0x7f2b201423f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:35.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.622+0000 7f2b27ca9640 10 monclient(hunting): picked mon.noname-a con 0x7f2b2013f500 addr v2:172.21.15.3:3300/0 2023-10-30T12:32:35.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.622+0000 7f2b27ca9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:35.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.622+0000 7f2b27ca9640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b200679a0 con 0x7f2b2013f500 2023-10-30T12:32:35.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.623+0000 7f2b27ca9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:35.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.623+0000 7f2b27ca9640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b200b1060 con 0x7f2b200a8e40 2023-10-30T12:32:35.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.623+0000 7f2b27ca9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:35.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.623+0000 7f2b27ca9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b200a9220 con 0x7f2b200b13c0 2023-10-30T12:32:35.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.623+0000 7f2b27ca9640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:35.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.623+0000 7f2b27ca9640 10 monclient(hunting): authenticate will time out at 2746.837081s 2023-10-30T12:32:35.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.623+0000 7f2b26246640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2b2013f500 0x7f2b201423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:35.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.623+0000 7f2b25244640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b200b13c0 0x7f2b20141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:35.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.623+0000 7f2b25a45640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2b200a8e40 0x7f2b2013efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:35.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.623+0000 7f2b26246640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2b2013f500 0x7f2b201423f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40464/0 (socket says 172.21.15.3:40464) 2023-10-30T12:32:35.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.623+0000 7f2b26246640 1 -- 172.21.15.3:0/2377477493 learned_addr learned my addr 172.21.15.3:0/2377477493 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:35.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.623+0000 7f2b26246640 10 monclient(hunting): get_auth_request con 0x7f2b2013f500 auth_method 0 2023-10-30T12:32:35.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.623+0000 7f2b26246640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:35.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.623+0000 7f2b26246640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:35.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.623+0000 7f2b26246640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:35.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b25244640 10 monclient(hunting): get_auth_request con 0x7f2b200b13c0 auth_method 0 2023-10-30T12:32:35.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b25244640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:35.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b25244640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:35.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b25244640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:35.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b25a45640 10 monclient(hunting): get_auth_request con 0x7f2b200a8e40 auth_method 0 2023-10-30T12:32:35.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b25a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:35.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b25a45640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:35.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b25a45640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:35.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b26246640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:35.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b26246640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:35.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b26246640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:35.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b25244640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:35.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b25244640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:35.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b25244640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:35.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b25a45640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:35.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b25a45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:35.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.624+0000 7f2b25a45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:35.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.626+0000 7f2b25244640 10 monclient(hunting): handle_auth_done global_id 15581 payload 274 2023-10-30T12:32:35.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.626+0000 7f2b25244640 1 -- 172.21.15.3:0/2377477493 >> v2:172.21.15.92:3300/0 conn(0x7f2b200a8e40 msgr2=0x7f2b2013efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:35.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.626+0000 7f2b25244640 1 --2- 172.21.15.3:0/2377477493 >> v2:172.21.15.92:3300/0 conn(0x7f2b200a8e40 0x7f2b2013efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.626+0000 7f2b25244640 1 -- 172.21.15.3:0/2377477493 >> v2:172.21.15.3:3300/0 conn(0x7f2b2013f500 msgr2=0x7f2b201423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:35.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.626+0000 7f2b25244640 1 --2- 172.21.15.3:0/2377477493 >> v2:172.21.15.3:3300/0 conn(0x7f2b2013f500 0x7f2b201423f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.626+0000 7f2b25244640 10 monclient: _finish_hunting 0 2023-10-30T12:32:35.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.626+0000 7f2b25244640 1 monclient: found mon.noname-b 2023-10-30T12:32:35.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.626+0000 7f2b25244640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:35.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.626+0000 7f2b25244640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:32:35.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.626+0000 7f2b25244640 1 -- 172.21.15.3:0/2377477493 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b200a8920 con 0x7f2b200b13c0 2023-10-30T12:32:35.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.628+0000 7f2b25244640 1 --2- 172.21.15.3:0/2377477493 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b200b13c0 0x7f2b20141eb0 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f2b200b8030 tx=0x7f2b1002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f674c7867fdf4f51 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:35.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b24a43640 1 -- 172.21.15.3:0/2377477493 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1002e050 con 0x7f2b200b13c0 2023-10-30T12:32:35.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b24a43640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:35.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b24a43640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:35.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b24a43640 10 monclient: dump: 2023-10-30T12:32:35.647 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:35.648 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:35.648 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.648 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.648 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:35.649 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:35.649 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:35.649 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:35.649 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:35.649 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:35.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b24a43640 10 monclient: _finish_auth 0 2023-10-30T12:32:35.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b24a43640 10 monclient: _check_auth_tickets 2023-10-30T12:32:35.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b24a43640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:35.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b24a43640 1 -- 172.21.15.3:0/2377477493 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b10030710 con 0x7f2b200b13c0 2023-10-30T12:32:35.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b27ca9640 5 monclient: authenticate success, global_id 15581 2023-10-30T12:32:35.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b27ca9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:32:35.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b24a43640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:35.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b27ca9640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:35.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b27ca9640 4 set_mon_vals no callback set 2023-10-30T12:32:35.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b24a43640 1 -- 172.21.15.3:0/2377477493 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b10030ab0 con 0x7f2b200b13c0 2023-10-30T12:32:35.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b24a43640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:35.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b24a43640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:32:35.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b24a43640 10 monclient: dump: 2023-10-30T12:32:35.652 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:35.653 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:35.653 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.653 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.653 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:35.653 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:35.654 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:35.654 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:35.654 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:35.654 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:35.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b27ca9640 10 monclient: shutdown 2023-10-30T12:32:35.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b27ca9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:35.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b27ca9640 1 -- 172.21.15.3:0/2377477493 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b200b13c0 msgr2=0x7f2b20141eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:35.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b27ca9640 1 --2- 172.21.15.3:0/2377477493 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b200b13c0 0x7f2b20141eb0 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f2b200b8030 tx=0x7f2b1002b710 comp rx=0 tx=0).stop 2023-10-30T12:32:35.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b27ca9640 1 -- 172.21.15.3:0/2377477493 shutdown_connections 2023-10-30T12:32:35.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b27ca9640 1 --2- 172.21.15.3:0/2377477493 >> v2:172.21.15.3:3300/0 conn(0x7f2b2013f500 0x7f2b201423f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b27ca9640 1 --2- 172.21.15.3:0/2377477493 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b200b13c0 0x7f2b20141eb0 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b27ca9640 1 --2- 172.21.15.3:0/2377477493 >> v2:172.21.15.92:3300/0 conn(0x7f2b200a8e40 0x7f2b2013efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.632+0000 7f2b27ca9640 1 -- 172.21.15.3:0/2377477493 >> 172.21.15.3:0/2377477493 conn(0x7f2b200b7180 msgr2=0x7f2b200b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:35.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.633+0000 7f2b27ca9640 1 -- 172.21.15.3:0/2377477493 shutdown_connections 2023-10-30T12:32:35.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.633+0000 7f2b27ca9640 1 -- 172.21.15.3:0/2377477493 wait complete. 2023-10-30T12:32:35.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.634+0000 7f2b27ca9640 10 monclient: build_initial_monmap 2023-10-30T12:32:35.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.634+0000 7f2b27ca9640 10 monclient: monmap: 2023-10-30T12:32:35.657 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:35.657 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:35.658 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:35.658 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:35.658 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:35.658 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:35.658 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:35.659 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:35.659 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:35.659 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:35.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.634+0000 7f2b27ca9640 1 librados: starting msgr at 2023-10-30T12:32:35.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.634+0000 7f2b27ca9640 1 librados: starting objecter 2023-10-30T12:32:35.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.634+0000 7f2b27ca9640 1 Processor -- start 2023-10-30T12:32:35.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.634+0000 7f2b27ca9640 1 -- start start 2023-10-30T12:32:35.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.634+0000 7f2b27ca9640 1 librados: setting wanted keys 2023-10-30T12:32:35.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.634+0000 7f2b27ca9640 1 librados: calling monclient init 2023-10-30T12:32:35.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.634+0000 7f2b27ca9640 10 monclient: init 2023-10-30T12:32:35.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:35.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:32:35.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b200a8e40 0x7f2b20145640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:35.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 10 monclient(hunting): picked mon.noname-a con 0x7f2b200a8e40 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:35.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b20145b80 0x7f2b20142d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:35.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 10 monclient(hunting): picked mon.noname-c con 0x7f2b20145b80 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:35.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b201432c0 0x7f2b201da1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:35.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 10 monclient(hunting): picked mon.noname-b con 0x7f2b201432c0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:35.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:35.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b200a83d0 con 0x7f2b200a8e40 2023-10-30T12:32:35.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:35.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2013dfb0 con 0x7f2b201432c0 2023-10-30T12:32:35.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:35.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b20144eb0 con 0x7f2b20145b80 2023-10-30T12:32:35.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:35.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b27ca9640 10 monclient(hunting): authenticate will time out at 2746.849219s 2023-10-30T12:32:35.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b25244640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b20145b80 0x7f2b20142d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:35.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b25a45640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b200a8e40 0x7f2b20145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:35.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b25244640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b20145b80 0x7f2b20142d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:37664/0 (socket says 172.21.15.3:37664) 2023-10-30T12:32:35.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b25244640 1 -- 172.21.15.3:0/1910584883 learned_addr learned my addr 172.21.15.3:0/1910584883 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:35.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b25244640 10 monclient(hunting): get_auth_request con 0x7f2b20145b80 auth_method 0 2023-10-30T12:32:35.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b25244640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:35.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b25244640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:35.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b25244640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:35.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b26246640 1 --2- 172.21.15.3:0/1910584883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b201432c0 0x7f2b201da1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:35.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b25a45640 10 monclient(hunting): get_auth_request con 0x7f2b200a8e40 auth_method 0 2023-10-30T12:32:35.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b25a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:35.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b25a45640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:35.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.635+0000 7f2b25a45640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:35.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.636+0000 7f2b26246640 10 monclient(hunting): get_auth_request con 0x7f2b201432c0 auth_method 0 2023-10-30T12:32:35.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.636+0000 7f2b26246640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:35.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.636+0000 7f2b26246640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:35.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.636+0000 7f2b26246640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:35.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.636+0000 7f2b25244640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:35.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.636+0000 7f2b25244640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:35.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.636+0000 7f2b25244640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:35.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.636+0000 7f2b25a45640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:35.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.636+0000 7f2b25a45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:35.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.636+0000 7f2b25a45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:35.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.636+0000 7f2b26246640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:35.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.636+0000 7f2b26246640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:35.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.636+0000 7f2b26246640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:35.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.639+0000 7f2b25a45640 10 monclient(hunting): handle_auth_done global_id 16662 payload 824 2023-10-30T12:32:35.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.639+0000 7f2b25a45640 1 -- 172.21.15.3:0/1910584883 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b20145b80 msgr2=0x7f2b20142d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:35.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.639+0000 7f2b25a45640 1 --2- 172.21.15.3:0/1910584883 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b20145b80 0x7f2b20142d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.639+0000 7f2b25a45640 1 -- 172.21.15.3:0/1910584883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b201432c0 msgr2=0x7f2b201da1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:35.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.639+0000 7f2b25a45640 1 --2- 172.21.15.3:0/1910584883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b201432c0 0x7f2b201da1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:35.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.639+0000 7f2b25a45640 10 monclient: _finish_hunting 0 2023-10-30T12:32:35.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.639+0000 7f2b25a45640 1 monclient: found mon.noname-a 2023-10-30T12:32:35.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.639+0000 7f2b25a45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:35.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.639+0000 7f2b25a45640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:32:35.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.639+0000 7f2b25a45640 1 -- 172.21.15.3:0/1910584883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2013d2d0 con 0x7f2b200a8e40 2023-10-30T12:32:35.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.639+0000 7f2b26246640 1 --2- 172.21.15.3:0/1910584883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b201432c0 0x7f2b201da1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:32:35.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.641+0000 7f2b25a45640 1 --2- 172.21.15.3:0/1910584883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b200a8e40 0x7f2b20145640 secure :-1 s=READY pgs=3611 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c00e870 tx=0x7f2b1c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:35.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.641+0000 7f2b167fc640 10 client.?.objecter ms_handle_connect 0x7f2b200a8e40 2023-10-30T12:32:35.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.641+0000 7f2b167fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:35.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b167fc640 1 -- 172.21.15.3:0/1910584883 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c00c940 con 0x7f2b200a8e40 2023-10-30T12:32:35.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:35.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:32:35.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b167fc640 10 monclient: dump: 2023-10-30T12:32:35.674 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:35.674 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:35.674 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.675 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.675 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:35.675 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:35.675 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:35.676 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:35.676 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:35.676 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:35.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b167fc640 10 monclient: _finish_auth 0 2023-10-30T12:32:35.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b167fc640 10 monclient: _check_auth_tickets 2023-10-30T12:32:35.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:35.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b167fc640 1 -- 172.21.15.3:0/1910584883 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1c00d040 con 0x7f2b200a8e40 2023-10-30T12:32:35.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b27ca9640 5 monclient: authenticate success, global_id 16662 2023-10-30T12:32:35.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b27ca9640 10 monclient: _renew_subs 2023-10-30T12:32:35.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b27ca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:32:35.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b27ca9640 1 -- 172.21.15.3:0/1910584883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b20143d00 con 0x7f2b200a8e40 2023-10-30T12:32:35.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b167fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:35.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b167fc640 1 -- 172.21.15.3:0/1910584883 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c00ccb0 con 0x7f2b200a8e40 2023-10-30T12:32:35.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:35.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:32:35.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b167fc640 10 monclient: dump: 2023-10-30T12:32:35.679 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:35.679 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:35.679 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.680 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:35.680 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:35.680 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:35.680 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:35.681 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:35.681 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:35.681 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:35.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b17fff640 4 set_mon_vals no callback set 2023-10-30T12:32:35.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b27ca9640 10 client.16662.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:35.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b27ca9640 10 monclient: _renew_subs 2023-10-30T12:32:35.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b27ca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:32:35.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b27ca9640 1 -- 172.21.15.3:0/1910584883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b20143920 con 0x7f2b200a8e40 2023-10-30T12:32:35.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.645+0000 7f2b27ca9640 1 librados: init done 2023-10-30T12:32:35.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.646+0000 7f2b27ca9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:35.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.646+0000 7f2b27ca9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:35.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.646+0000 7f2b27ca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:32:35.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.646+0000 7f2b27ca9640 1 -- 172.21.15.3:0/1910584883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b200b1b00 con 0x7f2b200a8e40 2023-10-30T12:32:35.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.665+0000 7f2b167fc640 1 -- 172.21.15.3:0/1910584883 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f2b1c013470 con 0x7f2b200a8e40 2023-10-30T12:32:35.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.665+0000 7f2b167fc640 1 --2- 172.21.15.3:0/1910584883 >> v2:172.21.15.92:6800/1750155735 conn(0x7f2b0406c7c0 0x7f2b0406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:35.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.666+0000 7f2b25244640 1 --2- 172.21.15.3:0/1910584883 >> v2:172.21.15.92:6800/1750155735 conn(0x7f2b0406c7c0 0x7f2b0406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:35.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.666+0000 7f2b25244640 10 monclient: get_auth_request con 0x7f2b0406c7c0 auth_method 0 2023-10-30T12:32:35.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.666+0000 7f2b25244640 1 --2- 172.21.15.3:0/1910584883 >> v2:172.21.15.92:6800/1750155735 conn(0x7f2b0406c7c0 0x7f2b0406ec80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2b1002b750 tx=0x7f2b1002f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:35.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.666+0000 7f2b167fc640 10 client.16662.objecter ms_handle_connect 0x7f2b0406c7c0 2023-10-30T12:32:35.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.931+0000 7f2b167fc640 1 -- 172.21.15.3:0/1910584883 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f2b1c00ce20 con 0x7f2b200a8e40 2023-10-30T12:32:35.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.931+0000 7f2b167fc640 10 client.16662.objecter ms_dispatch 0x7f2b20000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:35.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.931+0000 7f2b167fc640 3 client.16662.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:35.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.931+0000 7f2b167fc640 3 client.16662.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:35.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.931+0000 7f2b167fc640 20 client.16662.objecter dump_active .. 0 homeless 2023-10-30T12:32:35.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.931+0000 7f2b167fc640 1 -- 172.21.15.3:0/1910584883 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2b1c058610 con 0x7f2b200a8e40 2023-10-30T12:32:35.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.931+0000 7f2b167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:35.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:35.931+0000 7f2b167fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:32:36.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.108+0000 7f2b27ca9640 1 -- 172.21.15.3:0/1910584883 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b200a97b0 con 0x7f2b0406c7c0 2023-10-30T12:32:36.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.109+0000 7f2b167fc640 1 -- 172.21.15.3:0/1910584883 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2b10040070 con 0x7f2b0406c7c0 2023-10-30T12:32:36.111 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 3 to deep-scrub 2023-10-30T12:32:36.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 10 librados: watch_flush enter 2023-10-30T12:32:36.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 10 librados: watch_flush exit 2023-10-30T12:32:36.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 20 client.16662.objecter shutdown clearing up homeless session... 2023-10-30T12:32:36.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 10 client.16662.objecter successfully canceled tick 2023-10-30T12:32:36.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 1 -- 172.21.15.3:0/1910584883 >> v2:172.21.15.92:6800/1750155735 conn(0x7f2b0406c7c0 msgr2=0x7f2b0406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:36.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 1 --2- 172.21.15.3:0/1910584883 >> v2:172.21.15.92:6800/1750155735 conn(0x7f2b0406c7c0 0x7f2b0406ec80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2b1002b750 tx=0x7f2b1002f000 comp rx=0 tx=0).stop 2023-10-30T12:32:36.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 10 monclient: shutdown 2023-10-30T12:32:36.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:36.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 1 -- 172.21.15.3:0/1910584883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b200a8e40 msgr2=0x7f2b20145640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:36.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 1 --2- 172.21.15.3:0/1910584883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b200a8e40 0x7f2b20145640 secure :-1 s=READY pgs=3611 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c00e870 tx=0x7f2b1c00ed30 comp rx=0 tx=0).stop 2023-10-30T12:32:36.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 1 -- 172.21.15.3:0/1910584883 shutdown_connections 2023-10-30T12:32:36.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 1 --2- 172.21.15.3:0/1910584883 >> v2:172.21.15.92:6800/1750155735 conn(0x7f2b0406c7c0 0x7f2b0406ec80 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 1 --2- 172.21.15.3:0/1910584883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b201432c0 0x7f2b201da1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 1 --2- 172.21.15.3:0/1910584883 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b20145b80 0x7f2b20142d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 1 --2- 172.21.15.3:0/1910584883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b200a8e40 0x7f2b20145640 unknown :-1 s=CLOSED pgs=3611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 1 -- 172.21.15.3:0/1910584883 >> 172.21.15.3:0/1910584883 conn(0x7f2b200b7180 msgr2=0x7f2b200b4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:36.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 1 -- 172.21.15.3:0/1910584883 shutdown_connections 2023-10-30T12:32:36.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 1 -- 172.21.15.3:0/1910584883 wait complete. 2023-10-30T12:32:36.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.111+0000 7f2b27ca9640 1 librados: shutdown 2023-10-30T12:32:36.124 INFO:tasks.ceph:Scrubbing osd.4 2023-10-30T12:32:36.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:32:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.232+0000 7f3bfd264640 10 monclient: get_monmap_and_config 2023-10-30T12:32:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.232+0000 7f3bfd264640 10 monclient: build_initial_monmap 2023-10-30T12:32:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.233+0000 7f3bfd264640 10 monclient: monmap: 2023-10-30T12:32:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:36.236 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:36.233977+0000 2023-10-30T12:32:36.236 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:36.233977+0000 2023-10-30T12:32:36.236 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:36.236 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:36.236 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:36.237 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:36.237 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:36.237 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:36.237 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:36.237 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.233+0000 7f3bfd264640 1 Processor -- start 2023-10-30T12:32:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.233+0000 7f3bfd264640 1 -- start start 2023-10-30T12:32:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.233+0000 7f3bfd264640 10 monclient: init 2023-10-30T12:32:36.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bfd264640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:36.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bfd264640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2023-10-30T12:32:36.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bfd264640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3bf80a8850 addr v1:172.21.15.3:6789/0 2023-10-30T12:32:36.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bfd264640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3bf80b1b90 0x7f3bf80a9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bfd264640 10 monclient(hunting): picked mon.noname-b con 0x7f3bf80b1b90 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bfd264640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3bf813eec0 0x7f3bf813f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bfd264640 10 monclient(hunting): picked mon.noname-c con 0x7f3bf813eec0 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bfd264640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf80b1fd0 con 0x7f3bf80a8850 2023-10-30T12:32:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bfd264640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:36.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bfd264640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf8074040 con 0x7f3bf813eec0 2023-10-30T12:32:36.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bfd264640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:36.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bfd264640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf80a8c30 con 0x7f3bf80b1b90 2023-10-30T12:32:36.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bfd264640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:36.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bfd264640 10 monclient(hunting): authenticate will time out at 2747.448347s 2023-10-30T12:32:36.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bf6ffd640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3bf80a8850 0x7f3bf8142380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:48076/0 (socket says 172.21.15.3:48076) 2023-10-30T12:32:36.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bf6ffd640 1 -- 172.21.15.3:0/2314842785 learned_addr learned my addr 172.21.15.3:0/2314842785 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:32:36.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.234+0000 7f3bf67fc640 1 --2- 172.21.15.3:0/2314842785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3bf80b1b90 0x7f3bf80a9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:36.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.235+0000 7f3bf77fe640 1 --2- 172.21.15.3:0/2314842785 >> v2:172.21.15.92:3300/0 conn(0x7f3bf813eec0 0x7f3bf813f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:36.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.235+0000 7f3bf67fc640 10 monclient(hunting): get_auth_request con 0x7f3bf80b1b90 auth_method 0 2023-10-30T12:32:36.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.235+0000 7f3bf67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:36.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.235+0000 7f3bf67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:36.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.235+0000 7f3bf67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:36.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.235+0000 7f3bf77fe640 10 monclient(hunting): get_auth_request con 0x7f3bf813eec0 auth_method 0 2023-10-30T12:32:36.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.235+0000 7f3bf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:36.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.235+0000 7f3bf77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:36.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.235+0000 7f3bf77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:36.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.235+0000 7f3bf67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:36.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.235+0000 7f3bf67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:36.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.235+0000 7f3bf67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:36.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.235+0000 7f3bf77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:36.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.235+0000 7f3bf77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:36.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.235+0000 7f3bf77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:36.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.237+0000 7f3bf5ffb640 1 -- 172.21.15.3:0/2314842785 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194188310 0 0) 0x7f3bf0001490 con 0x7f3bf80a8850 2023-10-30T12:32:36.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.237+0000 7f3bf5ffb640 10 monclient(hunting): _negotiate 2023-10-30T12:32:36.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.237+0000 7f3bf5ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.237+0000 7f3bf5ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.237+0000 7f3bf5ffb640 10 monclient(hunting): my global_id is 16665 2023-10-30T12:32:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.237+0000 7f3bf5ffb640 1 -- 172.21.15.3:0/2314842785 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be0003520 con 0x7f3bf80a8850 2023-10-30T12:32:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.237+0000 7f3bf67fc640 10 monclient(hunting): handle_auth_done global_id 15587 payload 274 2023-10-30T12:32:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.237+0000 7f3bf67fc640 1 -- 172.21.15.3:0/2314842785 >> v2:172.21.15.92:3300/0 conn(0x7f3bf813eec0 msgr2=0x7f3bf813f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.237+0000 7f3bf67fc640 1 --2- 172.21.15.3:0/2314842785 >> v2:172.21.15.92:3300/0 conn(0x7f3bf813eec0 0x7f3bf813f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.237+0000 7f3bf67fc640 1 -- 172.21.15.3:0/2314842785 >> v1:172.21.15.3:6789/0 conn(0x7f3bf80a8850 legacy=0x7f3bf8142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.237+0000 7f3bf67fc640 10 monclient: _finish_hunting 0 2023-10-30T12:32:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.237+0000 7f3bf67fc640 1 monclient: found mon.noname-b 2023-10-30T12:32:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.237+0000 7f3bf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:36.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.237+0000 7f3bf67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:32:36.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.237+0000 7f3bf67fc640 1 -- 172.21.15.3:0/2314842785 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf80aa7b0 con 0x7f3bf80b1b90 2023-10-30T12:32:36.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.239+0000 7f3bf67fc640 1 --2- 172.21.15.3:0/2314842785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3bf80b1b90 0x7f3bf80a9350 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0097a0 tx=0x7f3bec02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4ba920af6ad42e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:36.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bf5ffb640 1 -- 172.21.15.3:0/2314842785 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec0315a0 con 0x7f3bf80b1b90 2023-10-30T12:32:36.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:36.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bf5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:36.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bf5ffb640 10 monclient: dump: 2023-10-30T12:32:36.255 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:36.255 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:36.255 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.255 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:36.257 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:36.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bf5ffb640 10 monclient: _finish_auth 0 2023-10-30T12:32:36.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bf5ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:32:36.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bf5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:36.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bf5ffb640 1 -- 172.21.15.3:0/2314842785 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bec031710 con 0x7f3bf80b1b90 2023-10-30T12:32:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bf5ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bf5ffb640 1 -- 172.21.15.3:0/2314842785 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec031ab0 con 0x7f3bf80b1b90 2023-10-30T12:32:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bfd264640 5 monclient: authenticate success, global_id 15587 2023-10-30T12:32:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bfd264640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bfd264640 4 set_mon_vals no callback set 2023-10-30T12:32:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bf5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:32:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bf5ffb640 10 monclient: dump: 2023-10-30T12:32:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:36.261 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bfd264640 10 monclient: shutdown 2023-10-30T12:32:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bfd264640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bfd264640 1 -- 172.21.15.3:0/2314842785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3bf80b1b90 msgr2=0x7f3bf80a9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bfd264640 1 --2- 172.21.15.3:0/2314842785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3bf80b1b90 0x7f3bf80a9350 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0097a0 tx=0x7f3bec02b840 comp rx=0 tx=0).stop 2023-10-30T12:32:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bfd264640 1 -- 172.21.15.3:0/2314842785 shutdown_connections 2023-10-30T12:32:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bfd264640 1 --2- 172.21.15.3:0/2314842785 >> v2:172.21.15.92:3300/0 conn(0x7f3bf813eec0 0x7f3bf813f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bfd264640 1 --2- 172.21.15.3:0/2314842785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3bf80b1b90 0x7f3bf80a9350 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bfd264640 1 -- 172.21.15.3:0/2314842785 >> 172.21.15.3:0/2314842785 conn(0x7f3bf80b7210 msgr2=0x7f3bf80b75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:36.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bfd264640 1 -- 172.21.15.3:0/2314842785 shutdown_connections 2023-10-30T12:32:36.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.243+0000 7f3bfd264640 1 -- 172.21.15.3:0/2314842785 wait complete. 2023-10-30T12:32:36.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.244+0000 7f3bfd264640 10 monclient: build_initial_monmap 2023-10-30T12:32:36.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.244+0000 7f3bfd264640 10 monclient: monmap: 2023-10-30T12:32:36.263 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:36.264 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:36.264 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:36.264 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:36.264 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:36.264 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:36.265 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:36.265 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:36.265 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:36.265 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:36.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.244+0000 7f3bfd264640 1 librados: starting msgr at 2023-10-30T12:32:36.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.244+0000 7f3bfd264640 1 librados: starting objecter 2023-10-30T12:32:36.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 1 Processor -- start 2023-10-30T12:32:36.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 1 -- start start 2023-10-30T12:32:36.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 1 librados: setting wanted keys 2023-10-30T12:32:36.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 1 librados: calling monclient init 2023-10-30T12:32:36.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 10 monclient: init 2023-10-30T12:32:36.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:36.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:32:36.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3bf80a8850 0x7f3bf81cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:36.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 10 monclient(hunting): picked mon.noname-b con 0x7f3bf80a8850 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:36.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3bf80b1b90 0x7f3bf81394d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:36.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 10 monclient(hunting): picked mon.noname-c con 0x7f3bf80b1b90 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:36.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bf813eec0 0x7f3bf81373b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:36.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 10 monclient(hunting): picked mon.noname-a con 0x7f3bf813eec0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:36.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:36.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf80aa180 con 0x7f3bf813eec0 2023-10-30T12:32:36.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:36.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf8139cd0 con 0x7f3bf80a8850 2023-10-30T12:32:36.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:36.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf813e5f0 con 0x7f3bf80b1b90 2023-10-30T12:32:36.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:36.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.245+0000 7f3bfd264640 10 monclient(hunting): authenticate will time out at 2747.459868s 2023-10-30T12:32:36.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf67fc640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3bf80b1b90 0x7f3bf81394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:36.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf77fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bf813eec0 0x7f3bf81373b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:36.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf6ffd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3bf80a8850 0x7f3bf81cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:36.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf67fc640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3bf80b1b90 0x7f3bf81394d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:37678/0 (socket says 172.21.15.3:37678) 2023-10-30T12:32:36.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf67fc640 1 -- 172.21.15.3:0/1888845852 learned_addr learned my addr 172.21.15.3:0/1888845852 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:36.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf67fc640 10 monclient(hunting): get_auth_request con 0x7f3bf80b1b90 auth_method 0 2023-10-30T12:32:36.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:36.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:36.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:36.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf77fe640 10 monclient(hunting): get_auth_request con 0x7f3bf813eec0 auth_method 0 2023-10-30T12:32:36.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:36.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:36.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:36.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf6ffd640 10 monclient(hunting): get_auth_request con 0x7f3bf80a8850 auth_method 0 2023-10-30T12:32:36.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:36.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:36.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.246+0000 7f3bf6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:36.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.247+0000 7f3bf67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:36.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.247+0000 7f3bf67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:36.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.247+0000 7f3bf67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:36.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.247+0000 7f3bf77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:36.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.247+0000 7f3bf77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:36.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.247+0000 7f3bf77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:36.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.247+0000 7f3bf6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:36.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.247+0000 7f3bf6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:36.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.247+0000 7f3bf6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:36.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.249+0000 7f3bf67fc640 10 monclient(hunting): handle_auth_done global_id 15590 payload 824 2023-10-30T12:32:36.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.249+0000 7f3bf67fc640 1 -- 172.21.15.3:0/1888845852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3bf80a8850 msgr2=0x7f3bf81cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:36.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.249+0000 7f3bf67fc640 1 --2- 172.21.15.3:0/1888845852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3bf80a8850 0x7f3bf81cca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.249+0000 7f3bf67fc640 1 -- 172.21.15.3:0/1888845852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bf813eec0 msgr2=0x7f3bf81373b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.249+0000 7f3bf67fc640 1 --2- 172.21.15.3:0/1888845852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bf813eec0 0x7f3bf81373b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.249+0000 7f3bf67fc640 10 monclient: _finish_hunting 0 2023-10-30T12:32:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.249+0000 7f3bf67fc640 1 monclient: found mon.noname-c 2023-10-30T12:32:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.249+0000 7f3bf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.249+0000 7f3bf67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.3:3301/0 2023-10-30T12:32:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.249+0000 7f3bf67fc640 1 -- 172.21.15.3:0/1888845852 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf813de20 con 0x7f3bf80b1b90 2023-10-30T12:32:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.251+0000 7f3bf67fc640 1 --2- 172.21.15.3:0/1888845852 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3bf80b1b90 0x7f3bf81394d0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f3bec034040 tx=0x7f3bec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.252+0000 7f3bdf7fe640 10 client.?.objecter ms_handle_connect 0x7f3bf80b1b90 2023-10-30T12:32:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.252+0000 7f3bdf7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.257+0000 7f3bdf7fe640 1 -- 172.21.15.3:0/1888845852 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec032840 con 0x7f3bf80b1b90 2023-10-30T12:32:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bdf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:32:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bdf7fe640 10 monclient: dump: 2023-10-30T12:32:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:36.281 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:36.281 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:36.281 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:36.281 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:36.281 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:36.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bdf7fe640 10 monclient: _finish_auth 0 2023-10-30T12:32:36.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bdf7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:32:36.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:36.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bdf7fe640 1 -- 172.21.15.3:0/1888845852 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bec03f030 con 0x7f3bf80b1b90 2023-10-30T12:32:36.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bfd264640 5 monclient: authenticate success, global_id 15590 2023-10-30T12:32:36.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bdf7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:36.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bdf7fe640 1 -- 172.21.15.3:0/1888845852 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec032bf0 con 0x7f3bf80b1b90 2023-10-30T12:32:36.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:36.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bdf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:32:36.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bdf7fe640 10 monclient: dump: 2023-10-30T12:32:36.284 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:36.284 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:36.284 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.284 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.284 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:36.285 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:36.285 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:36.285 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:36.285 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:36.285 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:36.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bfd264640 10 monclient: _renew_subs 2023-10-30T12:32:36.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bfd264640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:32:36.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bfd264640 1 -- 172.21.15.3:0/1888845852 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf813e020 con 0x7f3bf80b1b90 2023-10-30T12:32:36.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bf57fa640 4 set_mon_vals no callback set 2023-10-30T12:32:36.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bfd264640 10 client.15590.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:36.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bfd264640 10 monclient: _renew_subs 2023-10-30T12:32:36.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bfd264640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:32:36.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bfd264640 1 -- 172.21.15.3:0/1888845852 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf8137ad0 con 0x7f3bf80b1b90 2023-10-30T12:32:36.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.258+0000 7f3bfd264640 1 librados: init done 2023-10-30T12:32:36.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.259+0000 7f3bfd264640 10 client.15590.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:36.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.259+0000 7f3bfd264640 15 client.15590.objecter _session_command_op_assign -1 1 2023-10-30T12:32:36.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.259+0000 7f3bfd264640 20 client.15590.objecter _get_session osd=-1 returning homeless 2023-10-30T12:32:36.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.259+0000 7f3bfd264640 10 client.15590.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:36.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.259+0000 7f3bfd264640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.3:3301/0 2023-10-30T12:32:36.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.259+0000 7f3bfd264640 1 -- 172.21.15.3:0/1888845852 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3bf8141b80 con 0x7f3bf80b1b90 2023-10-30T12:32:36.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.284+0000 7f3bdf7fe640 1 -- 172.21.15.3:0/1888845852 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f3bec04e430 con 0x7f3bf80b1b90 2023-10-30T12:32:36.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.284+0000 7f3bdf7fe640 1 --2- 172.21.15.3:0/1888845852 >> v2:172.21.15.92:6800/1750155735 conn(0x7f3be406c7c0 0x7f3be406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:36.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.284+0000 7f3bf6ffd640 1 --2- 172.21.15.3:0/1888845852 >> v2:172.21.15.92:6800/1750155735 conn(0x7f3be406c7c0 0x7f3be406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:36.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.284+0000 7f3bf6ffd640 10 monclient: get_auth_request con 0x7f3be406c7c0 auth_method 0 2023-10-30T12:32:36.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.285+0000 7f3bf6ffd640 1 --2- 172.21.15.3:0/1888845852 >> v2:172.21.15.92:6800/1750155735 conn(0x7f3be406c7c0 0x7f3be406ec80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0001700 tx=0x7f3bf0001a90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:36.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.285+0000 7f3bdf7fe640 10 client.15590.objecter ms_handle_connect 0x7f3be406c7c0 2023-10-30T12:32:36.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 1 -- 172.21.15.3:0/1888845852 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f3bec079cf0 con 0x7f3bf80b1b90 2023-10-30T12:32:36.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 10 client.15590.objecter ms_dispatch 0x7f3bf8000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:36.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 3 client.15590.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:36.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 3 client.15590.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:36.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 10 client.15590.objecter checking command 1 2023-10-30T12:32:36.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 1 --2- 172.21.15.3:0/1888845852 >> v2:172.21.15.92:6801/4229938171 conn(0x7f3be4073a30 0x7f3be4075ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:36.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 20 client.15590.objecter _get_session s=0x7f3bec032bf0 osd=4 3 2023-10-30T12:32:36.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 20 client.15590.objecter put_session s=0x7f3bec032bf0 osd=4 3 2023-10-30T12:32:36.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 15 client.15590.objecter _session_command_op_remove -1 1 2023-10-30T12:32:36.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 20 client.15590.objecter _get_session s=0x7f3bec032bf0 osd=4 3 2023-10-30T12:32:36.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 20 client.15590.objecter get_session s=0x7f3bec032bf0 osd=4 3 2023-10-30T12:32:36.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 15 client.15590.objecter _session_command_op_assign 4 1 2023-10-30T12:32:36.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 20 client.15590.objecter put_session s=0x7f3bec032bf0 osd=4 4 2023-10-30T12:32:36.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 10 client.15590.objecter _send_command 1 2023-10-30T12:32:36.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 1 -- 172.21.15.3:0/1888845852 --> v2:172.21.15.92:6801/4229938171 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3bec032840 con 0x7f3be4073a30 2023-10-30T12:32:36.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 20 client.15590.objecter dump_active .. 0 homeless 2023-10-30T12:32:36.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 1 -- 172.21.15.3:0/1888845852 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3bec02e070 con 0x7f3bf80b1b90 2023-10-30T12:32:36.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bdf7fe640 10 monclient: handle_get_version_reply finishing 1 version 900 2023-10-30T12:32:36.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.287+0000 7f3bf77fe640 1 --2- 172.21.15.3:0/1888845852 >> v2:172.21.15.92:6801/4229938171 conn(0x7f3be4073a30 0x7f3be4075ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:36.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.288+0000 7f3bf77fe640 10 monclient: get_auth_request con 0x7f3be4073a30 auth_method 0 2023-10-30T12:32:36.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.290+0000 7f3bdf7fe640 10 client.15590.objecter ms_handle_connect 0x7f3be4073a30 2023-10-30T12:32:36.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.290+0000 7f3bf77fe640 1 --2- 172.21.15.3:0/1888845852 >> v2:172.21.15.92:6801/4229938171 conn(0x7f3be4073a30 0x7f3be4075ed0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:36.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.312+0000 7f3bdf7fe640 1 -- 172.21.15.3:0/1888845852 <== osd.4 v2:172.21.15.92:6801/4229938171 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f3bec032840 con 0x7f3be4073a30 2023-10-30T12:32:36.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.312+0000 7f3bdf7fe640 10 client.15590.objecter ms_dispatch 0x7f3bf8000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:32:36.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.312+0000 7f3bdf7fe640 10 client.15590.objecter _finish_command 1 = system:0 2023-10-30T12:32:36.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.312+0000 7f3bdf7fe640 20 client.15590.objecter put_session s=0x7f3bec032bf0 osd=4 4 2023-10-30T12:32:36.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.312+0000 7f3bdf7fe640 15 client.15590.objecter _session_command_op_remove 4 1 2023-10-30T12:32:36.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.325+0000 7f3bfd264640 10 client.15590.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:32:36.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.325+0000 7f3bfd264640 15 client.15590.objecter _session_command_op_assign -1 2 2023-10-30T12:32:36.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.325+0000 7f3bfd264640 20 client.15590.objecter _get_session s=0x7f3bec032bf0 osd=4 3 2023-10-30T12:32:36.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.325+0000 7f3bfd264640 20 client.15590.objecter put_session s=0x7f3bec032bf0 osd=4 3 2023-10-30T12:32:36.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.325+0000 7f3bfd264640 20 client.15590.objecter _get_session s=0x7f3bec032bf0 osd=4 3 2023-10-30T12:32:36.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.325+0000 7f3bfd264640 15 client.15590.objecter _session_command_op_remove -1 2 2023-10-30T12:32:36.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.325+0000 7f3bfd264640 20 client.15590.objecter get_session s=0x7f3bec032bf0 osd=4 3 2023-10-30T12:32:36.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.325+0000 7f3bfd264640 15 client.15590.objecter _session_command_op_assign 4 2 2023-10-30T12:32:36.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.325+0000 7f3bfd264640 20 client.15590.objecter put_session s=0x7f3bec032bf0 osd=4 4 2023-10-30T12:32:36.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.325+0000 7f3bfd264640 10 client.15590.objecter _send_command 2 2023-10-30T12:32:36.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.325+0000 7f3bfd264640 1 -- 172.21.15.3:0/1888845852 --> v2:172.21.15.92:6801/4229938171 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3bf8142a90 con 0x7f3be4073a30 2023-10-30T12:32:36.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bdf7fe640 1 -- 172.21.15.3:0/1888845852 <== osd.4 v2:172.21.15.92:6801/4229938171 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f3be8009b80 con 0x7f3be4073a30 2023-10-30T12:32:36.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bdf7fe640 10 client.15590.objecter ms_dispatch 0x7f3bf8000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:32:36.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bdf7fe640 10 client.15590.objecter _finish_command 2 = system:0 2023-10-30T12:32:36.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bdf7fe640 20 client.15590.objecter put_session s=0x7f3bec032bf0 osd=4 4 2023-10-30T12:32:36.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bdf7fe640 15 client.15590.objecter _session_command_op_remove 4 2 2023-10-30T12:32:36.341 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-10-30T12:32:36.341 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:32:36.341 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-10-30T12:32:36.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 10 librados: watch_flush enter 2023-10-30T12:32:36.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 10 librados: watch_flush exit 2023-10-30T12:32:36.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 10 client.15590.objecter close_session for osd.4 2023-10-30T12:32:36.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 1 -- 172.21.15.3:0/1888845852 >> v2:172.21.15.92:6801/4229938171 conn(0x7f3be4073a30 msgr2=0x7f3be4075ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:36.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 1 --2- 172.21.15.3:0/1888845852 >> v2:172.21.15.92:6801/4229938171 conn(0x7f3be4073a30 0x7f3be4075ed0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 20 client.15590.objecter put_session s=0x7f3bec032bf0 osd=4 1 2023-10-30T12:32:36.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 20 client.15590.objecter shutdown clearing up homeless session... 2023-10-30T12:32:36.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 10 client.15590.objecter successfully canceled tick 2023-10-30T12:32:36.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 1 -- 172.21.15.3:0/1888845852 >> v2:172.21.15.92:6800/1750155735 conn(0x7f3be406c7c0 msgr2=0x7f3be406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:36.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 1 --2- 172.21.15.3:0/1888845852 >> v2:172.21.15.92:6800/1750155735 conn(0x7f3be406c7c0 0x7f3be406ec80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0001700 tx=0x7f3bf0001a90 comp rx=0 tx=0).stop 2023-10-30T12:32:36.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 10 monclient: shutdown 2023-10-30T12:32:36.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:36.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 1 -- 172.21.15.3:0/1888845852 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3bf80b1b90 msgr2=0x7f3bf81394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:36.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 1 --2- 172.21.15.3:0/1888845852 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3bf80b1b90 0x7f3bf81394d0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f3bec034040 tx=0x7f3bec0046c0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bf77fe640 1 -- 172.21.15.3:0/1888845852 reap_dead start 2023-10-30T12:32:36.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 1 -- 172.21.15.3:0/1888845852 shutdown_connections 2023-10-30T12:32:36.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 1 -- 172.21.15.3:0/1888845852 >> 172.21.15.3:0/1888845852 conn(0x7f3bf80b7210 msgr2=0x7f3bf81380e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:36.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 1 -- 172.21.15.3:0/1888845852 shutdown_connections 2023-10-30T12:32:36.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 1 -- 172.21.15.3:0/1888845852 wait complete. 2023-10-30T12:32:36.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.339+0000 7f3bfd264640 1 librados: shutdown 2023-10-30T12:32:36.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-30T12:32:36.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.461+0000 7f000aee7640 10 monclient: get_monmap_and_config 2023-10-30T12:32:36.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.461+0000 7f000aee7640 10 monclient: build_initial_monmap 2023-10-30T12:32:36.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.461+0000 7f000aee7640 10 monclient: monmap: 2023-10-30T12:32:36.463 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:36.463 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:36.464 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:36.462313+0000 2023-10-30T12:32:36.464 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:36.462313+0000 2023-10-30T12:32:36.464 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:36.464 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:36.464 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:36.465 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:36.465 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:36.465 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:36.465 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:36.466 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:36.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.462+0000 7f000aee7640 1 Processor -- start 2023-10-30T12:32:36.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.462+0000 7f000aee7640 1 -- start start 2023-10-30T12:32:36.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.462+0000 7f000aee7640 10 monclient: init 2023-10-30T12:32:36.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.462+0000 7f000aee7640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:36.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.462+0000 7f000aee7640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-10-30T12:32:36.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.462+0000 7f000aee7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0004071760 addr v1:172.21.15.3:6789/0 2023-10-30T12:32:36.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.462+0000 7f000aee7640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0004071b40 0x7f000413bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:36.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.462+0000 7f000aee7640 10 monclient(hunting): picked mon.noname-c con 0x7f0004071b40 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:36.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.462+0000 7f000aee7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f000413e610 addr v1:172.21.15.92:6789/0 2023-10-30T12:32:36.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.462+0000 7f000aee7640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0004070fd0 con 0x7f0004071760 2023-10-30T12:32:36.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.462+0000 7f000aee7640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0004071240 con 0x7f000413e610 2023-10-30T12:32:36.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.462+0000 7f000aee7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:36.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.462+0000 7f000aee7640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0004067b90 con 0x7f0004071b40 2023-10-30T12:32:36.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.462+0000 7f000aee7640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:36.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.462+0000 7f000aee7640 10 monclient(hunting): authenticate will time out at 2747.676905s 2023-10-30T12:32:36.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.463+0000 7f0003fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0004071b40 0x7f000413bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:36.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.463+0000 7f0003fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0004071b40 0x7f000413bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44716/0 (socket says 172.21.15.3:44716) 2023-10-30T12:32:36.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.463+0000 7f0003fff640 1 -- 172.21.15.3:0/4288476206 learned_addr learned my addr 172.21.15.3:0/4288476206 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:36.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.463+0000 7f0003fff640 10 monclient(hunting): get_auth_request con 0x7f0004071b40 auth_method 0 2023-10-30T12:32:36.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.463+0000 7f0003fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:36.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.463+0000 7f0003fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:36.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.463+0000 7f0003fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:36.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.464+0000 7f0003fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:36.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.464+0000 7f0003fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:36.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.464+0000 7f0003fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:36.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.465+0000 7f0003fff640 10 monclient(hunting): handle_auth_done global_id 21859 payload 274 2023-10-30T12:32:36.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.465+0000 7f0003fff640 1 -- 172.21.15.3:0/4288476206 >> v1:172.21.15.92:6789/0 conn(0x7f000413e610 legacy=0x7f0004148f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:36.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.465+0000 7f0003fff640 1 -- 172.21.15.3:0/4288476206 >> v1:172.21.15.3:6789/0 conn(0x7f0004071760 legacy=0x7f00040acc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:36.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.465+0000 7f0003fff640 10 monclient: _finish_hunting 0 2023-10-30T12:32:36.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.465+0000 7f0003fff640 1 monclient: found mon.noname-c 2023-10-30T12:32:36.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.465+0000 7f0003fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:36.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.465+0000 7f0003fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:32:36.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.465+0000 7f0003fff640 1 -- 172.21.15.3:0/4288476206 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000413c090 con 0x7f0004071b40 2023-10-30T12:32:36.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.468+0000 7f0003fff640 1 --2- 172.21.15.3:0/4288476206 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0004071b40 0x7f000413bb50 secure :-1 s=READY pgs=5311 cs=0 l=1 rev1=1 crypto rx=0x7efff0009870 tx=0x7efff002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7250b7485923628 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:36.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f00037fe640 1 -- 172.21.15.3:0/4288476206 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff00317e0 con 0x7f0004071b40 2023-10-30T12:32:36.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f00037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:36.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f00037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:32:36.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f00037fe640 10 monclient: dump: 2023-10-30T12:32:36.483 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:36.483 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:36.483 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.484 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.484 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:36.484 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:36.484 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:36.484 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:36.485 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:36.485 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:36.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f00037fe640 10 monclient: _finish_auth 0 2023-10-30T12:32:36.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f00037fe640 10 monclient: _check_auth_tickets 2023-10-30T12:32:36.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f00037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:36.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f00037fe640 1 -- 172.21.15.3:0/4288476206 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff0031950 con 0x7f0004071b40 2023-10-30T12:32:36.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f00037fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:36.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f000aee7640 5 monclient: authenticate success, global_id 21859 2023-10-30T12:32:36.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f00037fe640 1 -- 172.21.15.3:0/4288476206 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff0031cf0 con 0x7f0004071b40 2023-10-30T12:32:36.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f000aee7640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:36.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f000aee7640 4 set_mon_vals no callback set 2023-10-30T12:32:36.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f00037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:36.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f00037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:36.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.471+0000 7f00037fe640 10 monclient: dump: 2023-10-30T12:32:36.488 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:36.488 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:36.488 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.488 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.488 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:36.488 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:36.489 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:36.489 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:36.489 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:36.489 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:36.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.472+0000 7f000aee7640 10 monclient: shutdown 2023-10-30T12:32:36.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.472+0000 7f000aee7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:36.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.472+0000 7f000aee7640 1 -- 172.21.15.3:0/4288476206 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0004071b40 msgr2=0x7f000413bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:36.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.472+0000 7f000aee7640 1 --2- 172.21.15.3:0/4288476206 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0004071b40 0x7f000413bb50 secure :-1 s=READY pgs=5311 cs=0 l=1 rev1=1 crypto rx=0x7efff0009870 tx=0x7efff002b880 comp rx=0 tx=0).stop 2023-10-30T12:32:36.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.472+0000 7f000aee7640 1 -- 172.21.15.3:0/4288476206 shutdown_connections 2023-10-30T12:32:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.472+0000 7f000aee7640 1 --2- 172.21.15.3:0/4288476206 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0004071b40 0x7f000413bb50 unknown :-1 s=CLOSED pgs=5311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.472+0000 7f000aee7640 1 -- 172.21.15.3:0/4288476206 >> 172.21.15.3:0/4288476206 conn(0x7f0004069d50 msgr2=0x7f000406a130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.472+0000 7f000aee7640 1 -- 172.21.15.3:0/4288476206 shutdown_connections 2023-10-30T12:32:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.472+0000 7f000aee7640 1 -- 172.21.15.3:0/4288476206 wait complete. 2023-10-30T12:32:36.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.473+0000 7f000aee7640 10 monclient: build_initial_monmap 2023-10-30T12:32:36.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.473+0000 7f000aee7640 10 monclient: monmap: 2023-10-30T12:32:36.492 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:36.492 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:36.492 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:36.492 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:36.493 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:36.493 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:36.493 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:36.493 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:36.493 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:36.494 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:36.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.473+0000 7f000aee7640 1 librados: starting msgr at 2023-10-30T12:32:36.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.473+0000 7f000aee7640 1 librados: starting objecter 2023-10-30T12:32:36.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.473+0000 7f000aee7640 1 Processor -- start 2023-10-30T12:32:36.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.473+0000 7f000aee7640 1 -- start start 2023-10-30T12:32:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.473+0000 7f000aee7640 1 librados: setting wanted keys 2023-10-30T12:32:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.473+0000 7f000aee7640 1 librados: calling monclient init 2023-10-30T12:32:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.473+0000 7f000aee7640 10 monclient: init 2023-10-30T12:32:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:32:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0004071760 0x7f0004139ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 10 monclient(hunting): picked mon.noname-c con 0x7f0004071760 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f000413e610 0x7f0004136900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 10 monclient(hunting): picked mon.noname-a con 0x7f000413e610 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:36.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0004131090 0x7f0004131520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:36.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 10 monclient(hunting): picked mon.noname-b con 0x7f0004131090 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:36.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:36.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000406f380 con 0x7f000413e610 2023-10-30T12:32:36.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:36.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000413a9c0 con 0x7f0004131090 2023-10-30T12:32:36.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:36.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000413a530 con 0x7f0004071760 2023-10-30T12:32:36.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:36.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f000aee7640 10 monclient(hunting): authenticate will time out at 2747.688483s 2023-10-30T12:32:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.474+0000 7f0009484640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0004131090 0x7f0004131520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.475+0000 7f0009484640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0004131090 0x7f0004131520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44728/0 (socket says 172.21.15.3:44728) 2023-10-30T12:32:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.475+0000 7f0009484640 1 -- 172.21.15.3:0/2776910727 learned_addr learned my addr 172.21.15.3:0/2776910727 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.475+0000 7f0009484640 10 monclient(hunting): get_auth_request con 0x7f0004131090 auth_method 0 2023-10-30T12:32:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.475+0000 7f0009484640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.475+0000 7f0009484640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.475+0000 7f0009484640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.475+0000 7f0009484640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.475+0000 7f0009484640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.475+0000 7f0009484640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:36.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.478+0000 7f0009484640 10 monclient(hunting): handle_auth_done global_id 21862 payload 824 2023-10-30T12:32:36.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.478+0000 7f0009484640 1 -- 172.21.15.3:0/2776910727 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0004071760 msgr2=0x7f0004139ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:36.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.478+0000 7f0009484640 1 --2- 172.21.15.3:0/2776910727 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0004071760 0x7f0004139ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.478+0000 7f0009484640 1 -- 172.21.15.3:0/2776910727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f000413e610 msgr2=0x7f0004136900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:36.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.478+0000 7f0009484640 1 --2- 172.21.15.3:0/2776910727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f000413e610 0x7f0004136900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.478+0000 7f0009484640 10 monclient: _finish_hunting 0 2023-10-30T12:32:36.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.478+0000 7f0009484640 1 monclient: found mon.noname-b 2023-10-30T12:32:36.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.478+0000 7f0009484640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:36.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.478+0000 7f0009484640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:36.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.478+0000 7f0009484640 1 -- 172.21.15.3:0/2776910727 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0004137630 con 0x7f0004131090 2023-10-30T12:32:36.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.480+0000 7f0009484640 1 --2- 172.21.15.3:0/2776910727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0004131090 0x7f0004131520 secure :-1 s=READY pgs=5312 cs=0 l=1 rev1=1 crypto rx=0x7efffc00b9c0 tx=0x7efffc00be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:36.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.480+0000 7f00017fa640 10 client.?.objecter ms_handle_connect 0x7f0004131090 2023-10-30T12:32:36.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.480+0000 7f00017fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:36.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f00017fa640 1 -- 172.21.15.3:0/2776910727 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc00cb40 con 0x7f0004131090 2023-10-30T12:32:36.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f00017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:36.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f00017fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:36.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f00017fa640 10 monclient: dump: 2023-10-30T12:32:36.505 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:36.506 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:36.506 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.506 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.506 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:36.507 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:36.507 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:36.507 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:36.507 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:36.507 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f00017fa640 10 monclient: _finish_auth 0 2023-10-30T12:32:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f00017fa640 10 monclient: _check_auth_tickets 2023-10-30T12:32:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f00017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f00017fa640 1 -- 172.21.15.3:0/2776910727 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efffc00ccb0 con 0x7f0004131090 2023-10-30T12:32:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f000aee7640 5 monclient: authenticate success, global_id 21862 2023-10-30T12:32:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f000aee7640 10 monclient: _renew_subs 2023-10-30T12:32:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f000aee7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f000aee7640 1 -- 172.21.15.3:0/2776910727 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0004139370 con 0x7f0004131090 2023-10-30T12:32:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f00017fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f00017fa640 1 -- 172.21.15.3:0/2776910727 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc014680 con 0x7f0004131090 2023-10-30T12:32:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f00017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f00017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f00017fa640 10 monclient: dump: 2023-10-30T12:32:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:36.513 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:36.514 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:36.514 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:36.514 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:36.514 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:36.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f0002ffd640 4 set_mon_vals no callback set 2023-10-30T12:32:36.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f000aee7640 10 client.21862.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:36.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f000aee7640 10 monclient: _renew_subs 2023-10-30T12:32:36.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f000aee7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:36.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f000aee7640 1 -- 172.21.15.3:0/2776910727 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00041d6ee0 con 0x7f0004131090 2023-10-30T12:32:36.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.483+0000 7f000aee7640 1 librados: init done 2023-10-30T12:32:36.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.485+0000 7f000aee7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:36.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.485+0000 7f000aee7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:36.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.485+0000 7f000aee7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:36.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.485+0000 7f000aee7640 1 -- 172.21.15.3:0/2776910727 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000404e9b0 con 0x7f0004131090 2023-10-30T12:32:36.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.728+0000 7f00017fa640 1 -- 172.21.15.3:0/2776910727 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7efffc01d3b0 con 0x7f0004131090 2023-10-30T12:32:36.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.729+0000 7f00017fa640 1 --2- 172.21.15.3:0/2776910727 >> v2:172.21.15.92:6800/1750155735 conn(0x7effe806c7c0 0x7effe806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:36.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.729+0000 7f0008c83640 1 --2- 172.21.15.3:0/2776910727 >> v2:172.21.15.92:6800/1750155735 conn(0x7effe806c7c0 0x7effe806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:36.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.729+0000 7f0008c83640 10 monclient: get_auth_request con 0x7effe806c7c0 auth_method 0 2023-10-30T12:32:36.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.729+0000 7f00017fa640 10 client.21862.objecter ms_handle_connect 0x7effe806c7c0 2023-10-30T12:32:36.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.729+0000 7f0008c83640 1 --2- 172.21.15.3:0/2776910727 >> v2:172.21.15.92:6800/1750155735 conn(0x7effe806c7c0 0x7effe806ec80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7efff8001560 tx=0x7efff8007990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:36.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.731+0000 7f00017fa640 1 -- 172.21.15.3:0/2776910727 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7efffc00ce30 con 0x7f0004131090 2023-10-30T12:32:36.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.731+0000 7f00017fa640 10 client.21862.objecter ms_dispatch 0x7f0004000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:36.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.731+0000 7f00017fa640 3 client.21862.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:36.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.731+0000 7f00017fa640 3 client.21862.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:36.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.731+0000 7f00017fa640 20 client.21862.objecter dump_active .. 0 homeless 2023-10-30T12:32:36.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.742+0000 7f00017fa640 1 -- 172.21.15.3:0/2776910727 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efffc058960 con 0x7f0004131090 2023-10-30T12:32:36.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.742+0000 7f00017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:36.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.742+0000 7f00017fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:32:36.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.946+0000 7f000aee7640 1 -- 172.21.15.3:0/2776910727 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f000413a680 con 0x7effe806c7c0 2023-10-30T12:32:36.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.947+0000 7f00017fa640 1 -- 172.21.15.3:0/2776910727 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f000413a680 con 0x7effe806c7c0 2023-10-30T12:32:36.949 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 4 to deep-scrub 2023-10-30T12:32:36.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.948+0000 7f000aee7640 10 librados: watch_flush enter 2023-10-30T12:32:36.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.948+0000 7f000aee7640 10 librados: watch_flush exit 2023-10-30T12:32:36.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.948+0000 7f000aee7640 20 client.21862.objecter shutdown clearing up homeless session... 2023-10-30T12:32:36.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.948+0000 7f000aee7640 10 client.21862.objecter successfully canceled tick 2023-10-30T12:32:36.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.949+0000 7f000aee7640 1 -- 172.21.15.3:0/2776910727 >> v2:172.21.15.92:6800/1750155735 conn(0x7effe806c7c0 msgr2=0x7effe806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:36.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.949+0000 7f000aee7640 1 --2- 172.21.15.3:0/2776910727 >> v2:172.21.15.92:6800/1750155735 conn(0x7effe806c7c0 0x7effe806ec80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7efff8001560 tx=0x7efff8007990 comp rx=0 tx=0).stop 2023-10-30T12:32:36.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.949+0000 7f000aee7640 10 monclient: shutdown 2023-10-30T12:32:36.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.949+0000 7f000aee7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:36.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.949+0000 7f000aee7640 1 -- 172.21.15.3:0/2776910727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0004131090 msgr2=0x7f0004131520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:36.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.949+0000 7f000aee7640 1 --2- 172.21.15.3:0/2776910727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0004131090 0x7f0004131520 secure :-1 s=READY pgs=5312 cs=0 l=1 rev1=1 crypto rx=0x7efffc00b9c0 tx=0x7efffc00be80 comp rx=0 tx=0).stop 2023-10-30T12:32:36.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.949+0000 7f000aee7640 1 -- 172.21.15.3:0/2776910727 shutdown_connections 2023-10-30T12:32:36.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.949+0000 7f000aee7640 1 --2- 172.21.15.3:0/2776910727 >> v2:172.21.15.92:6800/1750155735 conn(0x7effe806c7c0 0x7effe806ec80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.949+0000 7f000aee7640 1 --2- 172.21.15.3:0/2776910727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0004131090 0x7f0004131520 unknown :-1 s=CLOSED pgs=5312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.949+0000 7f000aee7640 1 --2- 172.21.15.3:0/2776910727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f000413e610 0x7f0004136900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.949+0000 7f000aee7640 1 --2- 172.21.15.3:0/2776910727 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0004071760 0x7f0004139ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:36.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.949+0000 7f000aee7640 1 -- 172.21.15.3:0/2776910727 >> 172.21.15.3:0/2776910727 conn(0x7f0004069d50 msgr2=0x7f00040aca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:36.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.949+0000 7f000aee7640 1 -- 172.21.15.3:0/2776910727 shutdown_connections 2023-10-30T12:32:36.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.949+0000 7f000aee7640 1 -- 172.21.15.3:0/2776910727 wait complete. 2023-10-30T12:32:36.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:36.949+0000 7f000aee7640 1 librados: shutdown 2023-10-30T12:32:36.961 INFO:tasks.ceph:Scrubbing osd.5 2023-10-30T12:32:36.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:32:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.070+0000 7fc6b1105640 10 monclient: get_monmap_and_config 2023-10-30T12:32:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.070+0000 7fc6b1105640 10 monclient: build_initial_monmap 2023-10-30T12:32:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.070+0000 7fc6b1105640 10 monclient: monmap: 2023-10-30T12:32:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:37.071646+0000 2023-10-30T12:32:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:37.071646+0000 2023-10-30T12:32:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:37.075 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:37.075 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:37.075 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.071+0000 7fc6b1105640 1 Processor -- start 2023-10-30T12:32:37.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.071+0000 7fc6b1105640 1 -- start start 2023-10-30T12:32:37.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.071+0000 7fc6b1105640 10 monclient: init 2023-10-30T12:32:37.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.071+0000 7fc6b1105640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:37.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6b1105640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-10-30T12:32:37.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6b1105640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc6ac142cd0 0x7fc6ac143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6b1105640 10 monclient(hunting): picked mon.noname-a con 0x7fc6ac142cd0 addr v2:172.21.15.3:3300/0 2023-10-30T12:32:37.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6b1105640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ac1430b0 0x7fc6ac0ad960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6b1105640 10 monclient(hunting): picked mon.noname-b con 0x7fc6ac1430b0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:37.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6b1105640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc6ac13cc00 0x7fc6ac13f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6b1105640 10 monclient(hunting): picked mon.noname-c con 0x7fc6ac13cc00 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:37.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6b1105640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6b1105640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6ac074040 con 0x7fc6ac142cd0 2023-10-30T12:32:37.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6b1105640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6b1105640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6ac0af030 con 0x7fc6ac13cc00 2023-10-30T12:32:37.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6b1105640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6b1105640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac071b10 con 0x7fc6ac1430b0 2023-10-30T12:32:37.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6b1105640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:37.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6b1105640 10 monclient(hunting): authenticate will time out at 2748.286166s 2023-10-30T12:32:37.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6aad9d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc6ac142cd0 0x7fc6ac143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6a3fff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ac1430b0 0x7fc6ac0ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6ab59e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc6ac13cc00 0x7fc6ac13f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6aad9d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc6ac142cd0 0x7fc6ac143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40520/0 (socket says 172.21.15.3:40520) 2023-10-30T12:32:37.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6aad9d640 1 -- 172.21.15.3:0/3605717682 learned_addr learned my addr 172.21.15.3:0/3605717682 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:37.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6aad9d640 10 monclient(hunting): get_auth_request con 0x7fc6ac142cd0 auth_method 0 2023-10-30T12:32:37.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6aad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.072+0000 7fc6aad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6a3fff640 10 monclient(hunting): get_auth_request con 0x7fc6ac1430b0 auth_method 0 2023-10-30T12:32:37.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6a3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6a3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6ab59e640 10 monclient(hunting): get_auth_request con 0x7fc6ac13cc00 auth_method 0 2023-10-30T12:32:37.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6ab59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6ab59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6ab59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6ab59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6ab59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.073+0000 7fc6ab59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.075+0000 7fc6aad9d640 10 monclient(hunting): handle_auth_done global_id 16671 payload 274 2023-10-30T12:32:37.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.075+0000 7fc6aad9d640 1 -- 172.21.15.3:0/3605717682 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ac1430b0 msgr2=0x7fc6ac0ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:37.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.075+0000 7fc6aad9d640 1 --2- 172.21.15.3:0/3605717682 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ac1430b0 0x7fc6ac0ad960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.075+0000 7fc6aad9d640 1 -- 172.21.15.3:0/3605717682 >> v2:172.21.15.92:3300/0 conn(0x7fc6ac13cc00 msgr2=0x7fc6ac13f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:37.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.075+0000 7fc6aad9d640 1 --2- 172.21.15.3:0/3605717682 >> v2:172.21.15.92:3300/0 conn(0x7fc6ac13cc00 0x7fc6ac13f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.075+0000 7fc6aad9d640 10 monclient: _finish_hunting 0 2023-10-30T12:32:37.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.075+0000 7fc6aad9d640 1 monclient: found mon.noname-a 2023-10-30T12:32:37.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.075+0000 7fc6aad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:37.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.075+0000 7fc6aad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:32:37.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.075+0000 7fc6aad9d640 1 -- 172.21.15.3:0/3605717682 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ac071f20 con 0x7fc6ac142cd0 2023-10-30T12:32:37.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.075+0000 7fc6ab59e640 1 --2- 172.21.15.3:0/3605717682 >> v2:172.21.15.92:3300/0 conn(0x7fc6ac13cc00 0x7fc6ac13f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:32:37.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.077+0000 7fc6aad9d640 1 --2- 172.21.15.3:0/3605717682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ac142cd0 0x7fc6ac143780 secure :-1 s=READY pgs=3613 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac06f8f0 tx=0x7fc69c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3f84923735c406 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6aa59c640 1 -- 172.21.15.3:0/3605717682 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c0317c0 con 0x7fc6ac142cd0 2023-10-30T12:32:37.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6aa59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:37.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6aa59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:32:37.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6aa59c640 10 monclient: dump: 2023-10-30T12:32:37.096 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:37.096 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:37.096 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.097 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.097 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:37.097 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:37.097 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:37.098 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:37.098 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:37.098 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6aa59c640 10 monclient: _finish_auth 0 2023-10-30T12:32:37.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6aa59c640 10 monclient: _check_auth_tickets 2023-10-30T12:32:37.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6aa59c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:37.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6aa59c640 1 -- 172.21.15.3:0/3605717682 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69c02e070 con 0x7fc6ac142cd0 2023-10-30T12:32:37.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6aa59c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:37.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6aa59c640 1 -- 172.21.15.3:0/3605717682 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c031b50 con 0x7fc6ac142cd0 2023-10-30T12:32:37.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6b1105640 5 monclient: authenticate success, global_id 16671 2023-10-30T12:32:37.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6b1105640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:37.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6b1105640 4 set_mon_vals no callback set 2023-10-30T12:32:37.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6aa59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:37.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6aa59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:32:37.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.080+0000 7fc6aa59c640 10 monclient: dump: 2023-10-30T12:32:37.101 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:37.101 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:37.101 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.102 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.102 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:37.102 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:37.102 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:37.103 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:37.103 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:37.103 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.081+0000 7fc6b1105640 10 monclient: shutdown 2023-10-30T12:32:37.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.081+0000 7fc6b1105640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:37.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.081+0000 7fc6b1105640 1 -- 172.21.15.3:0/3605717682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ac142cd0 msgr2=0x7fc6ac143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.081+0000 7fc6b1105640 1 --2- 172.21.15.3:0/3605717682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ac142cd0 0x7fc6ac143780 secure :-1 s=READY pgs=3613 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac06f8f0 tx=0x7fc69c02b860 comp rx=0 tx=0).stop 2023-10-30T12:32:37.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.081+0000 7fc6b1105640 1 -- 172.21.15.3:0/3605717682 shutdown_connections 2023-10-30T12:32:37.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.081+0000 7fc6b1105640 1 --2- 172.21.15.3:0/3605717682 >> v2:172.21.15.92:3300/0 conn(0x7fc6ac13cc00 0x7fc6ac13f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.081+0000 7fc6b1105640 1 --2- 172.21.15.3:0/3605717682 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ac1430b0 0x7fc6ac0ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.081+0000 7fc6b1105640 1 --2- 172.21.15.3:0/3605717682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ac142cd0 0x7fc6ac143780 unknown :-1 s=CLOSED pgs=3613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.081+0000 7fc6b1105640 1 -- 172.21.15.3:0/3605717682 >> 172.21.15.3:0/3605717682 conn(0x7fc6ac0afe50 msgr2=0x7fc6ac0b0260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:37.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.081+0000 7fc6b1105640 1 -- 172.21.15.3:0/3605717682 shutdown_connections 2023-10-30T12:32:37.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.081+0000 7fc6b1105640 1 -- 172.21.15.3:0/3605717682 wait complete. 2023-10-30T12:32:37.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.082+0000 7fc6b1105640 10 monclient: build_initial_monmap 2023-10-30T12:32:37.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.082+0000 7fc6b1105640 10 monclient: monmap: 2023-10-30T12:32:37.106 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:37.106 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:37.106 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:37.107 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:37.107 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:37.107 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:37.107 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:37.107 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:37.108 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:37.108 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 1 librados: starting msgr at 2023-10-30T12:32:37.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 1 librados: starting objecter 2023-10-30T12:32:37.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 1 Processor -- start 2023-10-30T12:32:37.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 1 -- start start 2023-10-30T12:32:37.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 1 librados: setting wanted keys 2023-10-30T12:32:37.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 1 librados: calling monclient init 2023-10-30T12:32:37.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 10 monclient: init 2023-10-30T12:32:37.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:37.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:32:37.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ac13cc00 0x7fc6ac13f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 10 monclient(hunting): picked mon.noname-c con 0x7fc6ac13cc00 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:37.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ac142cd0 0x7fc6ac1d7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 10 monclient(hunting): picked mon.noname-b con 0x7fc6ac142cd0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:37.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ac137c30 0x7fc6ac0a8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 10 monclient(hunting): picked mon.noname-a con 0x7fc6ac137c30 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:37.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac13d250 con 0x7fc6ac137c30 2023-10-30T12:32:37.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac1d84e0 con 0x7fc6ac142cd0 2023-10-30T12:32:37.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac1d8790 con 0x7fc6ac13cc00 2023-10-30T12:32:37.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:37.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.083+0000 7fc6b1105640 10 monclient(hunting): authenticate will time out at 2748.297952s 2023-10-30T12:32:37.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6ab59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ac137c30 0x7fc6ac0a8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6aad9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ac13cc00 0x7fc6ac13f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6ab59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ac137c30 0x7fc6ac0a8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40530/0 (socket says 172.21.15.3:40530) 2023-10-30T12:32:37.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6ab59e640 1 -- 172.21.15.3:0/2287526613 learned_addr learned my addr 172.21.15.3:0/2287526613 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:37.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6ab59e640 10 monclient(hunting): get_auth_request con 0x7fc6ac137c30 auth_method 0 2023-10-30T12:32:37.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6ab59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6ab59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6ab59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6a3fff640 1 --2- 172.21.15.3:0/2287526613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ac142cd0 0x7fc6ac1d7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6aad9d640 10 monclient(hunting): get_auth_request con 0x7fc6ac13cc00 auth_method 0 2023-10-30T12:32:37.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6aad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6aad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6a3fff640 10 monclient(hunting): get_auth_request con 0x7fc6ac142cd0 auth_method 0 2023-10-30T12:32:37.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6a3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.084+0000 7fc6a3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.085+0000 7fc6ab59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.085+0000 7fc6ab59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.085+0000 7fc6ab59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.085+0000 7fc6a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.085+0000 7fc6a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.085+0000 7fc6a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.085+0000 7fc6aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.085+0000 7fc6aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.085+0000 7fc6aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.088+0000 7fc6a3fff640 10 monclient(hunting): handle_auth_done global_id 21868 payload 824 2023-10-30T12:32:37.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.088+0000 7fc6a3fff640 1 -- 172.21.15.3:0/2287526613 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ac13cc00 msgr2=0x7fc6ac13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.088+0000 7fc6a3fff640 1 --2- 172.21.15.3:0/2287526613 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ac13cc00 0x7fc6ac13f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.088+0000 7fc6a3fff640 1 -- 172.21.15.3:0/2287526613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ac137c30 msgr2=0x7fc6ac0a8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.088+0000 7fc6a3fff640 1 --2- 172.21.15.3:0/2287526613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ac137c30 0x7fc6ac0a8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.088+0000 7fc6a3fff640 10 monclient: _finish_hunting 0 2023-10-30T12:32:37.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.088+0000 7fc6a3fff640 1 monclient: found mon.noname-b 2023-10-30T12:32:37.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.088+0000 7fc6a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:37.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.088+0000 7fc6a3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:37.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.088+0000 7fc6a3fff640 1 -- 172.21.15.3:0/2287526613 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ac0a91e0 con 0x7fc6ac142cd0 2023-10-30T12:32:37.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.090+0000 7fc6a3fff640 1 --2- 172.21.15.3:0/2287526613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ac142cd0 0x7fc6ac1d7fa0 secure :-1 s=READY pgs=5313 cs=0 l=1 rev1=1 crypto rx=0x7fc69400e840 tx=0x7fc69400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.090+0000 7fc6a37fe640 10 client.?.objecter ms_handle_connect 0x7fc6ac142cd0 2023-10-30T12:32:37.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.090+0000 7fc6a37fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:37.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6a37fe640 1 -- 172.21.15.3:0/2287526613 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69400c730 con 0x7fc6ac142cd0 2023-10-30T12:32:37.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:37.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:37.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6a37fe640 10 monclient: dump: 2023-10-30T12:32:37.125 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:37.125 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:37.126 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.126 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.126 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:37.126 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:37.127 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:37.127 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:37.127 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:37.127 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6a37fe640 10 monclient: _finish_auth 0 2023-10-30T12:32:37.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6a37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:32:37.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:37.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6b1105640 5 monclient: authenticate success, global_id 21868 2023-10-30T12:32:37.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6a37fe640 1 -- 172.21.15.3:0/2287526613 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69400d040 con 0x7fc6ac142cd0 2023-10-30T12:32:37.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6a37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:37.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6b1105640 10 monclient: _renew_subs 2023-10-30T12:32:37.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6a37fe640 1 -- 172.21.15.3:0/2287526613 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69400cac0 con 0x7fc6ac142cd0 2023-10-30T12:32:37.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6b1105640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:37.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6b1105640 1 -- 172.21.15.3:0/2287526613 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ac0a94a0 con 0x7fc6ac142cd0 2023-10-30T12:32:37.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:37.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:37.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6a37fe640 10 monclient: dump: 2023-10-30T12:32:37.130 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:37.130 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:37.133 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.133 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.134 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:37.134 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:37.134 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:37.134 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:37.135 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:37.135 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6a9d9b640 4 set_mon_vals no callback set 2023-10-30T12:32:37.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6b1105640 10 client.21868.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:37.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6b1105640 10 monclient: _renew_subs 2023-10-30T12:32:37.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6b1105640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:37.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6b1105640 1 -- 172.21.15.3:0/2287526613 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ac0a99c0 con 0x7fc6ac142cd0 2023-10-30T12:32:37.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.097+0000 7fc6b1105640 1 librados: init done 2023-10-30T12:32:37.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.098+0000 7fc6b1105640 10 client.21868.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:37.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.098+0000 7fc6b1105640 15 client.21868.objecter _session_command_op_assign -1 1 2023-10-30T12:32:37.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.098+0000 7fc6b1105640 20 client.21868.objecter _get_session osd=-1 returning homeless 2023-10-30T12:32:37.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.098+0000 7fc6b1105640 10 client.21868.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:37.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.098+0000 7fc6b1105640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:37.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.098+0000 7fc6b1105640 1 -- 172.21.15.3:0/2287526613 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc6ac0704b0 con 0x7fc6ac142cd0 2023-10-30T12:32:37.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.116+0000 7fc6a37fe640 1 -- 172.21.15.3:0/2287526613 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fc69400cc30 con 0x7fc6ac142cd0 2023-10-30T12:32:37.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.117+0000 7fc6a37fe640 1 --2- 172.21.15.3:0/2287526613 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc68c06c810 0x7fc68c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.117+0000 7fc6aad9d640 1 --2- 172.21.15.3:0/2287526613 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc68c06c810 0x7fc68c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.117+0000 7fc6aad9d640 10 monclient: get_auth_request con 0x7fc68c06c810 auth_method 0 2023-10-30T12:32:37.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.118+0000 7fc6aad9d640 1 --2- 172.21.15.3:0/2287526613 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc68c06c810 0x7fc68c06ecd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc69c036000 tx=0x7fc69c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.118+0000 7fc6a37fe640 10 client.21868.objecter ms_handle_connect 0x7fc68c06c810 2023-10-30T12:32:37.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 1 -- 172.21.15.3:0/2287526613 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7fc694058140 con 0x7fc6ac142cd0 2023-10-30T12:32:37.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 10 client.21868.objecter ms_dispatch 0x7fc6ac000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:37.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 3 client.21868.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:37.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 3 client.21868.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:37.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 10 client.21868.objecter checking command 1 2023-10-30T12:32:37.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 1 --2- 172.21.15.3:0/2287526613 >> v2:172.21.15.92:6805/3208162957 conn(0x7fc68c073a80 0x7fc68c075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 20 client.21868.objecter _get_session s=0x7fc69400cac0 osd=5 3 2023-10-30T12:32:37.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 20 client.21868.objecter put_session s=0x7fc69400cac0 osd=5 3 2023-10-30T12:32:37.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 15 client.21868.objecter _session_command_op_remove -1 1 2023-10-30T12:32:37.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 20 client.21868.objecter _get_session s=0x7fc69400cac0 osd=5 3 2023-10-30T12:32:37.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 20 client.21868.objecter get_session s=0x7fc69400cac0 osd=5 3 2023-10-30T12:32:37.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 15 client.21868.objecter _session_command_op_assign 5 1 2023-10-30T12:32:37.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 20 client.21868.objecter put_session s=0x7fc69400cac0 osd=5 4 2023-10-30T12:32:37.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 10 client.21868.objecter _send_command 1 2023-10-30T12:32:37.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 1 -- 172.21.15.3:0/2287526613 --> v2:172.21.15.92:6805/3208162957 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc69400c730 con 0x7fc68c073a80 2023-10-30T12:32:37.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 20 client.21868.objecter dump_active .. 0 homeless 2023-10-30T12:32:37.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 1 -- 172.21.15.3:0/2287526613 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc694054480 con 0x7fc6ac142cd0 2023-10-30T12:32:37.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.120+0000 7fc6a37fe640 10 monclient: handle_get_version_reply finishing 1 version 900 2023-10-30T12:32:37.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.121+0000 7fc6ab59e640 1 --2- 172.21.15.3:0/2287526613 >> v2:172.21.15.92:6805/3208162957 conn(0x7fc68c073a80 0x7fc68c075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.121+0000 7fc6ab59e640 10 monclient: get_auth_request con 0x7fc68c073a80 auth_method 0 2023-10-30T12:32:37.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.124+0000 7fc6ab59e640 1 --2- 172.21.15.3:0/2287526613 >> v2:172.21.15.92:6805/3208162957 conn(0x7fc68c073a80 0x7fc68c075f20 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.124+0000 7fc6a37fe640 10 client.21868.objecter ms_handle_connect 0x7fc68c073a80 2023-10-30T12:32:37.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.149+0000 7fc6a37fe640 1 -- 172.21.15.3:0/2287526613 <== osd.5 v2:172.21.15.92:6805/3208162957 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fc69400c730 con 0x7fc68c073a80 2023-10-30T12:32:37.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.149+0000 7fc6a37fe640 10 client.21868.objecter ms_dispatch 0x7fc6ac000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:32:37.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.149+0000 7fc6a37fe640 10 client.21868.objecter _finish_command 1 = system:0 2023-10-30T12:32:37.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.149+0000 7fc6a37fe640 20 client.21868.objecter put_session s=0x7fc69400cac0 osd=5 4 2023-10-30T12:32:37.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.149+0000 7fc6a37fe640 15 client.21868.objecter _session_command_op_remove 5 1 2023-10-30T12:32:37.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.162+0000 7fc6b1105640 10 client.21868.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:32:37.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.162+0000 7fc6b1105640 15 client.21868.objecter _session_command_op_assign -1 2 2023-10-30T12:32:37.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.162+0000 7fc6b1105640 20 client.21868.objecter _get_session s=0x7fc69400cac0 osd=5 3 2023-10-30T12:32:37.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.162+0000 7fc6b1105640 20 client.21868.objecter put_session s=0x7fc69400cac0 osd=5 3 2023-10-30T12:32:37.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.162+0000 7fc6b1105640 20 client.21868.objecter _get_session s=0x7fc69400cac0 osd=5 3 2023-10-30T12:32:37.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.162+0000 7fc6b1105640 15 client.21868.objecter _session_command_op_remove -1 2 2023-10-30T12:32:37.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.162+0000 7fc6b1105640 20 client.21868.objecter get_session s=0x7fc69400cac0 osd=5 3 2023-10-30T12:32:37.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.162+0000 7fc6b1105640 15 client.21868.objecter _session_command_op_assign 5 2 2023-10-30T12:32:37.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.162+0000 7fc6b1105640 20 client.21868.objecter put_session s=0x7fc69400cac0 osd=5 4 2023-10-30T12:32:37.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.162+0000 7fc6b1105640 10 client.21868.objecter _send_command 2 2023-10-30T12:32:37.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.162+0000 7fc6b1105640 1 -- 172.21.15.3:0/2287526613 --> v2:172.21.15.92:6805/3208162957 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc6ac0a94a0 con 0x7fc68c073a80 2023-10-30T12:32:37.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6a37fe640 1 -- 172.21.15.3:0/2287526613 <== osd.5 v2:172.21.15.92:6805/3208162957 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fc6ac0a94a0 con 0x7fc68c073a80 2023-10-30T12:32:37.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6a37fe640 10 client.21868.objecter ms_dispatch 0x7fc6ac000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:32:37.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6a37fe640 10 client.21868.objecter _finish_command 2 = system:0 2023-10-30T12:32:37.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6a37fe640 20 client.21868.objecter put_session s=0x7fc69400cac0 osd=5 4 2023-10-30T12:32:37.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6a37fe640 15 client.21868.objecter _session_command_op_remove 5 2 2023-10-30T12:32:37.179 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-10-30T12:32:37.180 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:32:37.180 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-10-30T12:32:37.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 10 librados: watch_flush enter 2023-10-30T12:32:37.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 10 librados: watch_flush exit 2023-10-30T12:32:37.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 10 client.21868.objecter close_session for osd.5 2023-10-30T12:32:37.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 1 -- 172.21.15.3:0/2287526613 >> v2:172.21.15.92:6805/3208162957 conn(0x7fc68c073a80 msgr2=0x7fc68c075f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 1 --2- 172.21.15.3:0/2287526613 >> v2:172.21.15.92:6805/3208162957 conn(0x7fc68c073a80 0x7fc68c075f20 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 20 client.21868.objecter put_session s=0x7fc69400cac0 osd=5 1 2023-10-30T12:32:37.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 20 client.21868.objecter shutdown clearing up homeless session... 2023-10-30T12:32:37.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 10 client.21868.objecter successfully canceled tick 2023-10-30T12:32:37.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 1 -- 172.21.15.3:0/2287526613 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc68c06c810 msgr2=0x7fc68c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 1 --2- 172.21.15.3:0/2287526613 >> v2:172.21.15.92:6800/1750155735 conn(0x7fc68c06c810 0x7fc68c06ecd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc69c036000 tx=0x7fc69c03d040 comp rx=0 tx=0).stop 2023-10-30T12:32:37.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 10 monclient: shutdown 2023-10-30T12:32:37.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:37.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 1 -- 172.21.15.3:0/2287526613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ac142cd0 msgr2=0x7fc6ac1d7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 1 --2- 172.21.15.3:0/2287526613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ac142cd0 0x7fc6ac1d7fa0 secure :-1 s=READY pgs=5313 cs=0 l=1 rev1=1 crypto rx=0x7fc69400e840 tx=0x7fc69400ed00 comp rx=0 tx=0).stop 2023-10-30T12:32:37.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6ab59e640 1 -- 172.21.15.3:0/2287526613 reap_dead start 2023-10-30T12:32:37.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 1 -- 172.21.15.3:0/2287526613 shutdown_connections 2023-10-30T12:32:37.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 1 -- 172.21.15.3:0/2287526613 >> 172.21.15.3:0/2287526613 conn(0x7fc6ac0afe50 msgr2=0x7fc6ac06a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:37.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 1 -- 172.21.15.3:0/2287526613 shutdown_connections 2023-10-30T12:32:37.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 1 -- 172.21.15.3:0/2287526613 wait complete. 2023-10-30T12:32:37.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.177+0000 7fc6b1105640 1 librados: shutdown 2023-10-30T12:32:37.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-30T12:32:37.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.296+0000 7fbb09402640 10 monclient: get_monmap_and_config 2023-10-30T12:32:37.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.296+0000 7fbb09402640 10 monclient: build_initial_monmap 2023-10-30T12:32:37.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.296+0000 7fbb09402640 10 monclient: monmap: 2023-10-30T12:32:37.299 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:37.299 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:37.299 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:37.297227+0000 2023-10-30T12:32:37.299 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:37.297227+0000 2023-10-30T12:32:37.300 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:37.300 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:37.300 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:37.300 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:37.300 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:37.301 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:37.301 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:37.301 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.296+0000 7fbb09402640 1 Processor -- start 2023-10-30T12:32:37.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.296+0000 7fbb09402640 1 -- start start 2023-10-30T12:32:37.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.296+0000 7fbb09402640 10 monclient: init 2023-10-30T12:32:37.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.297+0000 7fbb09402640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:37.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.297+0000 7fbb09402640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-10-30T12:32:37.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.297+0000 7fbb09402640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb040a8a50 addr v1:172.21.15.3:6789/0 2023-10-30T12:32:37.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.297+0000 7fbb09402640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbb040b3380 0x7fbb04141820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.297+0000 7fbb09402640 10 monclient(hunting): picked mon.noname-a con 0x7fbb040b3380 addr v2:172.21.15.3:3300/0 2023-10-30T12:32:37.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.297+0000 7fbb09402640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbb04141d60 0x7fbb04142160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.297+0000 7fbb09402640 10 monclient(hunting): picked mon.noname-c con 0x7fbb04141d60 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:37.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.297+0000 7fbb09402640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb040aa560 con 0x7fbb040a8a50 2023-10-30T12:32:37.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.297+0000 7fbb09402640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.297+0000 7fbb09402640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb04067810 con 0x7fbb040b3380 2023-10-30T12:32:37.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.297+0000 7fbb09402640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.297+0000 7fbb09402640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb040a8e30 con 0x7fbb04141d60 2023-10-30T12:32:37.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.297+0000 7fbb09402640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:37.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.297+0000 7fbb09402640 10 monclient(hunting): authenticate will time out at 2748.511578s 2023-10-30T12:32:37.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.298+0000 7fbb037fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbb04141d60 0x7fbb04142160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.298+0000 7fbb037fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbb04141d60 0x7fbb04142160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44762/0 (socket says 172.21.15.3:44762) 2023-10-30T12:32:37.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.298+0000 7fbb037fe640 1 -- 172.21.15.3:0/1968749814 learned_addr learned my addr 172.21.15.3:0/1968749814 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:37.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.298+0000 7fbb037fe640 10 monclient(hunting): get_auth_request con 0x7fbb04141d60 auth_method 0 2023-10-30T12:32:37.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.298+0000 7fbb037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.298+0000 7fbb037fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.298+0000 7fbb037fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.298+0000 7fbb027fc640 1 --2- 172.21.15.3:0/1968749814 >> v2:172.21.15.3:3300/0 conn(0x7fbb040b3380 0x7fbb04141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.298+0000 7fbb037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.298+0000 7fbb037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.298+0000 7fbb037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.299+0000 7fbb027fc640 10 monclient(hunting): get_auth_request con 0x7fbb040b3380 auth_method 0 2023-10-30T12:32:37.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.299+0000 7fbb027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.299+0000 7fbb027fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.299+0000 7fbb027fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.300+0000 7fbb027fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.300+0000 7fbb027fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.300+0000 7fbb027fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.300+0000 7fbb037fe640 10 monclient(hunting): handle_auth_done global_id 21871 payload 274 2023-10-30T12:32:37.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.300+0000 7fbb037fe640 1 -- 172.21.15.3:0/1968749814 >> v2:172.21.15.3:3300/0 conn(0x7fbb040b3380 msgr2=0x7fbb04141820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:37.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.300+0000 7fbb037fe640 1 --2- 172.21.15.3:0/1968749814 >> v2:172.21.15.3:3300/0 conn(0x7fbb040b3380 0x7fbb04141820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.300+0000 7fbb037fe640 1 -- 172.21.15.3:0/1968749814 >> v1:172.21.15.3:6789/0 conn(0x7fbb040a8a50 legacy=0x7fbb0413ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.300+0000 7fbb037fe640 10 monclient: _finish_hunting 0 2023-10-30T12:32:37.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.300+0000 7fbb037fe640 1 monclient: found mon.noname-c 2023-10-30T12:32:37.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.300+0000 7fbb037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:37.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.300+0000 7fbb037fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:32:37.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.300+0000 7fbb037fe640 1 -- 172.21.15.3:0/1968749814 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb040b37c0 con 0x7fbb04141d60 2023-10-30T12:32:37.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.302+0000 7fbb037fe640 1 --2- 172.21.15.3:0/1968749814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb04141d60 0x7fbb04142160 secure :-1 s=READY pgs=5316 cs=0 l=1 rev1=1 crypto rx=0x7fbb040a9bc0 tx=0x7fbafc02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bc1641efc253945 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.305+0000 7fbb01ffb640 1 -- 172.21.15.3:0/1968749814 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc031640 con 0x7fbb04141d60 2023-10-30T12:32:37.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.305+0000 7fbb01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:37.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.305+0000 7fbb01ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:32:37.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.305+0000 7fbb01ffb640 10 monclient: dump: 2023-10-30T12:32:37.318 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:37.319 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:37.319 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.319 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.319 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:37.319 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:37.320 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:37.320 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:37.320 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:37.320 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.305+0000 7fbb01ffb640 10 monclient: _finish_auth 0 2023-10-30T12:32:37.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.305+0000 7fbb01ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:32:37.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.305+0000 7fbb01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:37.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.305+0000 7fbb01ffb640 1 -- 172.21.15.3:0/1968749814 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbafc0317b0 con 0x7fbb04141d60 2023-10-30T12:32:37.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.305+0000 7fbb01ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:37.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.305+0000 7fbb01ffb640 1 -- 172.21.15.3:0/1968749814 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc031b50 con 0x7fbb04141d60 2023-10-30T12:32:37.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.305+0000 7fbb09402640 5 monclient: authenticate success, global_id 21871 2023-10-30T12:32:37.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.305+0000 7fbb09402640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:37.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.305+0000 7fbb09402640 4 set_mon_vals no callback set 2023-10-30T12:32:37.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.305+0000 7fbb01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:37.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.306+0000 7fbb01ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:37.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.306+0000 7fbb01ffb640 10 monclient: dump: 2023-10-30T12:32:37.323 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:37.323 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:37.323 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.324 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.324 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:37.324 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:37.324 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:37.324 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:37.325 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:37.325 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.306+0000 7fbb09402640 10 monclient: shutdown 2023-10-30T12:32:37.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.306+0000 7fbb09402640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:37.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.306+0000 7fbb09402640 1 -- 172.21.15.3:0/1968749814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb04141d60 msgr2=0x7fbb04142160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.306+0000 7fbb09402640 1 --2- 172.21.15.3:0/1968749814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb04141d60 0x7fbb04142160 secure :-1 s=READY pgs=5316 cs=0 l=1 rev1=1 crypto rx=0x7fbb040a9bc0 tx=0x7fbafc02b8f0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.306+0000 7fbb09402640 1 -- 172.21.15.3:0/1968749814 shutdown_connections 2023-10-30T12:32:37.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.306+0000 7fbb09402640 1 --2- 172.21.15.3:0/1968749814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb04141d60 0x7fbb04142160 unknown :-1 s=CLOSED pgs=5316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.306+0000 7fbb09402640 1 --2- 172.21.15.3:0/1968749814 >> v2:172.21.15.3:3300/0 conn(0x7fbb040b3380 0x7fbb04141820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.306+0000 7fbb09402640 1 -- 172.21.15.3:0/1968749814 >> 172.21.15.3:0/1968749814 conn(0x7fbb04065c90 msgr2=0x7fbb040660a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:37.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.306+0000 7fbb09402640 1 -- 172.21.15.3:0/1968749814 shutdown_connections 2023-10-30T12:32:37.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.306+0000 7fbb09402640 1 -- 172.21.15.3:0/1968749814 wait complete. 2023-10-30T12:32:37.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.307+0000 7fbb09402640 10 monclient: build_initial_monmap 2023-10-30T12:32:37.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.307+0000 7fbb09402640 10 monclient: monmap: 2023-10-30T12:32:37.327 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:37.328 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:37.328 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:37.328 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:37.328 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:37.329 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:37.329 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:37.329 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:37.329 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:37.330 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.307+0000 7fbb09402640 1 librados: starting msgr at 2023-10-30T12:32:37.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.307+0000 7fbb09402640 1 librados: starting objecter 2023-10-30T12:32:37.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.307+0000 7fbb09402640 1 Processor -- start 2023-10-30T12:32:37.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.307+0000 7fbb09402640 1 -- start start 2023-10-30T12:32:37.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.307+0000 7fbb09402640 1 librados: setting wanted keys 2023-10-30T12:32:37.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.307+0000 7fbb09402640 1 librados: calling monclient init 2023-10-30T12:32:37.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.307+0000 7fbb09402640 10 monclient: init 2023-10-30T12:32:37.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:37.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:32:37.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb040a8a50 0x7fbb040b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 10 monclient(hunting): picked mon.noname-b con 0x7fbb040a8a50 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:37.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb040b3380 0x7fbb041412b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 10 monclient(hunting): picked mon.noname-a con 0x7fbb040b3380 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:37.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb04141d60 0x7fbb041d6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 10 monclient(hunting): picked mon.noname-c con 0x7fbb04141d60 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:37.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb04141a30 con 0x7fbb040b3380 2023-10-30T12:32:37.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb041417f0 con 0x7fbb040a8a50 2023-10-30T12:32:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb040aaa10 con 0x7fbb04141d60 2023-10-30T12:32:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:37.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb09402640 10 monclient(hunting): authenticate will time out at 2748.522588s 2023-10-30T12:32:37.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.308+0000 7fbb027fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb040b3380 0x7fbb041412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb02ffd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb040a8a50 0x7fbb040b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb037fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb04141d60 0x7fbb041d6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb027fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb040b3380 0x7fbb041412b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40554/0 (socket says 172.21.15.3:40554) 2023-10-30T12:32:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb027fc640 1 -- 172.21.15.3:0/218529386 learned_addr learned my addr 172.21.15.3:0/218529386 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb027fc640 10 monclient(hunting): get_auth_request con 0x7fbb040b3380 auth_method 0 2023-10-30T12:32:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb027fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb027fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb02ffd640 10 monclient(hunting): get_auth_request con 0x7fbb040a8a50 auth_method 0 2023-10-30T12:32:37.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb02ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb02ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb037fe640 10 monclient(hunting): get_auth_request con 0x7fbb04141d60 auth_method 0 2023-10-30T12:32:37.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb037fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb037fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb027fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb027fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb027fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.309+0000 7fbb02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.310+0000 7fbb037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.310+0000 7fbb037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.310+0000 7fbb037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.312+0000 7fbb02ffd640 10 monclient(hunting): handle_auth_done global_id 21874 payload 824 2023-10-30T12:32:37.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.312+0000 7fbb02ffd640 1 -- 172.21.15.3:0/218529386 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb04141d60 msgr2=0x7fbb041d6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.312+0000 7fbb02ffd640 1 --2- 172.21.15.3:0/218529386 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb04141d60 0x7fbb041d6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.312+0000 7fbb02ffd640 1 -- 172.21.15.3:0/218529386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb040b3380 msgr2=0x7fbb041412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.312+0000 7fbb02ffd640 1 --2- 172.21.15.3:0/218529386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb040b3380 0x7fbb041412b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.312+0000 7fbb02ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:32:37.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.312+0000 7fbb02ffd640 1 monclient: found mon.noname-b 2023-10-30T12:32:37.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.312+0000 7fbb02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:37.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.312+0000 7fbb02ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:37.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.312+0000 7fbb02ffd640 1 -- 172.21.15.3:0/218529386 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb040aacf0 con 0x7fbb040a8a50 2023-10-30T12:32:37.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.314+0000 7fbb02ffd640 1 --2- 172.21.15.3:0/218529386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb040a8a50 0x7fbb040b1aa0 secure :-1 s=READY pgs=5317 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400c770 tx=0x7fbaf400cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.314+0000 7fbafb7fe640 10 client.?.objecter ms_handle_connect 0x7fbb040a8a50 2023-10-30T12:32:37.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.314+0000 7fbafb7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:37.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbafb7fe640 1 -- 172.21.15.3:0/218529386 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4003b70 con 0x7fbb040a8a50 2023-10-30T12:32:37.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbafb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:37.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbafb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:37.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbafb7fe640 10 monclient: dump: 2023-10-30T12:32:37.348 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:37.348 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:37.348 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.349 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.349 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:37.349 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:37.349 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:37.349 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:37.349 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:37.350 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbafb7fe640 10 monclient: _finish_auth 0 2023-10-30T12:32:37.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbafb7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:32:37.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbafb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:37.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbafb7fe640 1 -- 172.21.15.3:0/218529386 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf400e040 con 0x7fbb040a8a50 2023-10-30T12:32:37.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbb09402640 5 monclient: authenticate success, global_id 21874 2023-10-30T12:32:37.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbb09402640 10 monclient: _renew_subs 2023-10-30T12:32:37.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbb09402640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:37.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbb09402640 1 -- 172.21.15.3:0/218529386 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb040b21b0 con 0x7fbb040a8a50 2023-10-30T12:32:37.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbafb7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:37.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbafb7fe640 1 -- 172.21.15.3:0/218529386 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf400ce80 con 0x7fbb040a8a50 2023-10-30T12:32:37.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbafb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:37.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbafb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:37.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbafb7fe640 10 monclient: dump: 2023-10-30T12:32:37.353 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:37.353 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:37.353 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.353 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.353 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:37.354 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:37.354 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:37.354 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:37.354 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:37.355 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbb017fa640 4 set_mon_vals no callback set 2023-10-30T12:32:37.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbb09402640 10 client.21874.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:37.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbb09402640 10 monclient: _renew_subs 2023-10-30T12:32:37.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbb09402640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:37.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbb09402640 1 -- 172.21.15.3:0/218529386 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb041d7380 con 0x7fbb040a8a50 2023-10-30T12:32:37.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.318+0000 7fbb09402640 1 librados: init done 2023-10-30T12:32:37.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.319+0000 7fbb09402640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:37.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.319+0000 7fbb09402640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:37.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.319+0000 7fbb09402640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:37.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.319+0000 7fbb09402640 1 -- 172.21.15.3:0/218529386 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb0413c480 con 0x7fbb040a8a50 2023-10-30T12:32:37.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.336+0000 7fbafb7fe640 1 -- 172.21.15.3:0/218529386 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fbaf400a7f0 con 0x7fbb040a8a50 2023-10-30T12:32:37.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.337+0000 7fbafb7fe640 1 --2- 172.21.15.3:0/218529386 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbae406c7c0 0x7fbae406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.337+0000 7fbb027fc640 1 --2- 172.21.15.3:0/218529386 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbae406c7c0 0x7fbae406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.337+0000 7fbb027fc640 10 monclient: get_auth_request con 0x7fbae406c7c0 auth_method 0 2023-10-30T12:32:37.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.337+0000 7fbb027fc640 1 --2- 172.21.15.3:0/218529386 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbae406c7c0 0x7fbae406ec80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbb041d7cc0 tx=0x7fbaec009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.337+0000 7fbafb7fe640 10 client.21874.objecter ms_handle_connect 0x7fbae406c7c0 2023-10-30T12:32:37.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.339+0000 7fbafb7fe640 1 -- 172.21.15.3:0/218529386 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7fbaf4058580 con 0x7fbb040a8a50 2023-10-30T12:32:37.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.339+0000 7fbafb7fe640 10 client.21874.objecter ms_dispatch 0x7fbb04000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:37.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.339+0000 7fbafb7fe640 3 client.21874.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:37.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.339+0000 7fbafb7fe640 3 client.21874.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:37.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.339+0000 7fbafb7fe640 20 client.21874.objecter dump_active .. 0 homeless 2023-10-30T12:32:37.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.569+0000 7fbafb7fe640 1 -- 172.21.15.3:0/218529386 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbaf405c620 con 0x7fbb040a8a50 2023-10-30T12:32:37.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.569+0000 7fbafb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:37.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.569+0000 7fbafb7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:32:37.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.750+0000 7fbb09402640 1 -- 172.21.15.3:0/218529386 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb040b4f40 con 0x7fbae406c7c0 2023-10-30T12:32:37.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.750+0000 7fbafb7fe640 1 -- 172.21.15.3:0/218529386 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbb040b4f40 con 0x7fbae406c7c0 2023-10-30T12:32:37.752 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 5 to deep-scrub 2023-10-30T12:32:37.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.752+0000 7fbb09402640 10 librados: watch_flush enter 2023-10-30T12:32:37.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.752+0000 7fbb09402640 10 librados: watch_flush exit 2023-10-30T12:32:37.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.752+0000 7fbb09402640 20 client.21874.objecter shutdown clearing up homeless session... 2023-10-30T12:32:37.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.752+0000 7fbb09402640 10 client.21874.objecter successfully canceled tick 2023-10-30T12:32:37.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.752+0000 7fbb09402640 1 -- 172.21.15.3:0/218529386 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbae406c7c0 msgr2=0x7fbae406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.752+0000 7fbb09402640 1 --2- 172.21.15.3:0/218529386 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbae406c7c0 0x7fbae406ec80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbb041d7cc0 tx=0x7fbaec009190 comp rx=0 tx=0).stop 2023-10-30T12:32:37.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.752+0000 7fbb09402640 10 monclient: shutdown 2023-10-30T12:32:37.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.753+0000 7fbb09402640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:37.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.753+0000 7fbb09402640 1 -- 172.21.15.3:0/218529386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb040a8a50 msgr2=0x7fbb040b1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.753+0000 7fbb09402640 1 --2- 172.21.15.3:0/218529386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb040a8a50 0x7fbb040b1aa0 secure :-1 s=READY pgs=5317 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400c770 tx=0x7fbaf400cc30 comp rx=0 tx=0).stop 2023-10-30T12:32:37.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.753+0000 7fbb09402640 1 -- 172.21.15.3:0/218529386 shutdown_connections 2023-10-30T12:32:37.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.753+0000 7fbb09402640 1 --2- 172.21.15.3:0/218529386 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbae406c7c0 0x7fbae406ec80 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.753+0000 7fbb09402640 1 --2- 172.21.15.3:0/218529386 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb04141d60 0x7fbb041d6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.753+0000 7fbb09402640 1 --2- 172.21.15.3:0/218529386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb040b3380 0x7fbb041412b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.753+0000 7fbb09402640 1 --2- 172.21.15.3:0/218529386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb040a8a50 0x7fbb040b1aa0 unknown :-1 s=CLOSED pgs=5317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.753+0000 7fbb09402640 1 -- 172.21.15.3:0/218529386 >> 172.21.15.3:0/218529386 conn(0x7fbb04065c90 msgr2=0x7fbb04142140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:37.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.753+0000 7fbb09402640 1 -- 172.21.15.3:0/218529386 shutdown_connections 2023-10-30T12:32:37.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.753+0000 7fbb09402640 1 -- 172.21.15.3:0/218529386 wait complete. 2023-10-30T12:32:37.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.753+0000 7fbb09402640 1 librados: shutdown 2023-10-30T12:32:37.765 INFO:tasks.ceph:Scrubbing osd.6 2023-10-30T12:32:37.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:32:37.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.874+0000 7f1962234640 10 monclient: get_monmap_and_config 2023-10-30T12:32:37.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.874+0000 7f1962234640 10 monclient: build_initial_monmap 2023-10-30T12:32:37.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.874+0000 7f1962234640 10 monclient: monmap: 2023-10-30T12:32:37.876 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:37.876 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:37.877 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:37.875261+0000 2023-10-30T12:32:37.877 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:37.875261+0000 2023-10-30T12:32:37.877 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:37.877 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:37.877 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:37.878 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:37.878 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:37.878 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:37.878 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:37.878 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 1 Processor -- start 2023-10-30T12:32:37.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 1 -- start start 2023-10-30T12:32:37.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 10 monclient: init 2023-10-30T12:32:37.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:37.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-10-30T12:32:37.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f195c0b4420 0x7f195c13ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 10 monclient(hunting): picked mon.noname-b con 0x7f195c0b4420 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:37.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f195c0b12d0 0x7f195c13ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 10 monclient(hunting): picked mon.noname-c con 0x7f195c0b12d0 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:37.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f195c141910 addr v1:172.21.15.92:6789/0 2023-10-30T12:32:37.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195c0a9120 con 0x7f195c141910 2023-10-30T12:32:37.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195c067a10 con 0x7f195c0b12d0 2023-10-30T12:32:37.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195c0b4800 con 0x7f195c0b4420 2023-10-30T12:32:37.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:37.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.875+0000 7f1962234640 10 monclient(hunting): authenticate will time out at 2749.089840s 2023-10-30T12:32:37.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.876+0000 7f195affd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f195c0b12d0 0x7f195c13ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.876+0000 7f195b7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f195c0b4420 0x7f195c13ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.876+0000 7f195affd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f195c0b12d0 0x7f195c13ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44782/0 (socket says 172.21.15.3:44782) 2023-10-30T12:32:37.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.876+0000 7f195affd640 1 -- 172.21.15.3:0/178942639 learned_addr learned my addr 172.21.15.3:0/178942639 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:37.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.876+0000 7f195affd640 10 monclient(hunting): get_auth_request con 0x7f195c0b12d0 auth_method 0 2023-10-30T12:32:37.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.876+0000 7f195affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.876+0000 7f195affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.876+0000 7f195affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.877+0000 7f195b7fe640 10 monclient(hunting): get_auth_request con 0x7f195c0b4420 auth_method 0 2023-10-30T12:32:37.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.877+0000 7f195b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.877+0000 7f195b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.877+0000 7f195b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.877+0000 7f195affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.877+0000 7f195affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.877+0000 7f195affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.877+0000 7f195b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.877+0000 7f195b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.877+0000 7f195b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.878+0000 7f195affd640 10 monclient(hunting): handle_auth_done global_id 21877 payload 274 2023-10-30T12:32:37.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.878+0000 7f195affd640 1 -- 172.21.15.3:0/178942639 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f195c0b4420 msgr2=0x7f195c13ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:37.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.878+0000 7f195affd640 1 --2- 172.21.15.3:0/178942639 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f195c0b4420 0x7f195c13ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.878+0000 7f195affd640 1 -- 172.21.15.3:0/178942639 >> v1:172.21.15.92:6789/0 conn(0x7f195c141910 legacy=0x7f195c141da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:37.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.878+0000 7f195affd640 10 monclient: _finish_hunting 0 2023-10-30T12:32:37.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.878+0000 7f195affd640 1 monclient: found mon.noname-c 2023-10-30T12:32:37.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.878+0000 7f195affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:37.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.878+0000 7f195affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:32:37.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.878+0000 7f195affd640 1 -- 172.21.15.3:0/178942639 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c0aa460 con 0x7f195c0b12d0 2023-10-30T12:32:37.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.880+0000 7f195affd640 1 --2- 172.21.15.3:0/178942639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f195c0b12d0 0x7f195c13ef90 secure :-1 s=READY pgs=5318 cs=0 l=1 rev1=1 crypto rx=0x7f1950009870 tx=0x7f195002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0a80a72adeef2b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f195a7fc640 1 -- 172.21.15.3:0/178942639 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19500317c0 con 0x7f195c0b12d0 2023-10-30T12:32:37.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f195a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:37.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f195a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:32:37.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f195a7fc640 10 monclient: dump: 2023-10-30T12:32:37.893 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:37.895 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f195a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:32:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f195a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:32:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f195a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f195a7fc640 1 -- 172.21.15.3:0/178942639 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195002e070 con 0x7f195c0b12d0 2023-10-30T12:32:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f195a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f195a7fc640 1 -- 172.21.15.3:0/178942639 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950031b50 con 0x7f195c0b12d0 2023-10-30T12:32:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f1962234640 5 monclient: authenticate success, global_id 21877 2023-10-30T12:32:37.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f1962234640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:37.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f195a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:37.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f1962234640 4 set_mon_vals no callback set 2023-10-30T12:32:37.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f195a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:37.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.884+0000 7f195a7fc640 10 monclient: dump: 2023-10-30T12:32:37.898 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:37.898 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:37.898 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.898 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.898 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:37.899 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:37.899 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:37.899 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:37.899 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:37.899 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.885+0000 7f1962234640 10 monclient: shutdown 2023-10-30T12:32:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.885+0000 7f1962234640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.885+0000 7f1962234640 1 -- 172.21.15.3:0/178942639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f195c0b12d0 msgr2=0x7f195c13ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.885+0000 7f1962234640 1 --2- 172.21.15.3:0/178942639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f195c0b12d0 0x7f195c13ef90 secure :-1 s=READY pgs=5318 cs=0 l=1 rev1=1 crypto rx=0x7f1950009870 tx=0x7f195002b860 comp rx=0 tx=0).stop 2023-10-30T12:32:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.885+0000 7f1962234640 1 -- 172.21.15.3:0/178942639 shutdown_connections 2023-10-30T12:32:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.885+0000 7f1962234640 1 --2- 172.21.15.3:0/178942639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f195c0b12d0 0x7f195c13ef90 unknown :-1 s=CLOSED pgs=5318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.885+0000 7f1962234640 1 --2- 172.21.15.3:0/178942639 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f195c0b4420 0x7f195c13ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.885+0000 7f1962234640 1 -- 172.21.15.3:0/178942639 >> 172.21.15.3:0/178942639 conn(0x7f195c065ea0 msgr2=0x7f195c0662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.885+0000 7f1962234640 1 -- 172.21.15.3:0/178942639 shutdown_connections 2023-10-30T12:32:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.885+0000 7f1962234640 1 -- 172.21.15.3:0/178942639 wait complete. 2023-10-30T12:32:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.886+0000 7f1962234640 10 monclient: build_initial_monmap 2023-10-30T12:32:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.886+0000 7f1962234640 10 monclient: monmap: 2023-10-30T12:32:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:37.903 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:37.903 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:37.903 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:37.903 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:37.903 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:37.904 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:37.904 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.887+0000 7f1962234640 1 librados: starting msgr at 2023-10-30T12:32:37.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.887+0000 7f1962234640 1 librados: starting objecter 2023-10-30T12:32:37.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.887+0000 7f1962234640 1 Processor -- start 2023-10-30T12:32:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.887+0000 7f1962234640 1 -- start start 2023-10-30T12:32:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.887+0000 7f1962234640 1 librados: setting wanted keys 2023-10-30T12:32:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.887+0000 7f1962234640 1 librados: calling monclient init 2023-10-30T12:32:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.887+0000 7f1962234640 10 monclient: init 2023-10-30T12:32:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:37.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:32:37.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f195c0b12d0 0x7f195c1407b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 10 monclient(hunting): picked mon.noname-a con 0x7f195c0b12d0 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:37.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f195c0b4420 0x7f195c1d6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 10 monclient(hunting): picked mon.noname-c con 0x7f195c0b4420 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:37.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f195c141910 0x7f195c1d7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 10 monclient(hunting): picked mon.noname-b con 0x7f195c141910 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:37.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c13ef70 con 0x7f195c0b12d0 2023-10-30T12:32:37.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c141470 con 0x7f195c141910 2023-10-30T12:32:37.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:37.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195c1c9180 con 0x7f195c0b4420 2023-10-30T12:32:37.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.888+0000 7f1962234640 10 monclient(hunting): authenticate will time out at 2749.102810s 2023-10-30T12:32:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195affd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f195c0b4420 0x7f195c1d6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f195c0b12d0 0x7f195c1407b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195bfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f195c141910 0x7f195c1d7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f195c0b12d0 0x7f195c1407b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40564/0 (socket says 172.21.15.3:40564) 2023-10-30T12:32:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195b7fe640 1 -- 172.21.15.3:0/3378325361 learned_addr learned my addr 172.21.15.3:0/3378325361 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195b7fe640 10 monclient(hunting): get_auth_request con 0x7f195c0b12d0 auth_method 0 2023-10-30T12:32:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195affd640 10 monclient(hunting): get_auth_request con 0x7f195c0b4420 auth_method 0 2023-10-30T12:32:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195bfff640 10 monclient(hunting): get_auth_request con 0x7f195c141910 auth_method 0 2023-10-30T12:32:37.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:37.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:37.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.889+0000 7f195bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:37.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.890+0000 7f195affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.890+0000 7f195affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.890+0000 7f195affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.890+0000 7f195bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.890+0000 7f195bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.890+0000 7f195bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.890+0000 7f195b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:37.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.890+0000 7f195b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:37.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.890+0000 7f195b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:37.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.892+0000 7f195bfff640 10 monclient(hunting): handle_auth_done global_id 21880 payload 824 2023-10-30T12:32:37.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.892+0000 7f195bfff640 1 -- 172.21.15.3:0/3378325361 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f195c0b4420 msgr2=0x7f195c1d6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.892+0000 7f195bfff640 1 --2- 172.21.15.3:0/3378325361 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f195c0b4420 0x7f195c1d6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.892+0000 7f195bfff640 1 -- 172.21.15.3:0/3378325361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f195c0b12d0 msgr2=0x7f195c1407b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.892+0000 7f195bfff640 1 --2- 172.21.15.3:0/3378325361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f195c0b12d0 0x7f195c1407b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.892+0000 7f195bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:32:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.892+0000 7f195bfff640 1 monclient: found mon.noname-b 2023-10-30T12:32:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.892+0000 7f195bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.892+0000 7f195bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.892+0000 7f195bfff640 1 -- 172.21.15.3:0/3378325361 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c1c9350 con 0x7f195c141910 2023-10-30T12:32:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.895+0000 7f195bfff640 1 --2- 172.21.15.3:0/3378325361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f195c141910 0x7f195c1d7250 secure :-1 s=READY pgs=5319 cs=0 l=1 rev1=1 crypto rx=0x7f195400be20 tx=0x7f195400bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.895+0000 7f1943fff640 10 client.?.objecter ms_handle_connect 0x7f195c141910 2023-10-30T12:32:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.895+0000 7f1943fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1943fff640 1 -- 172.21.15.3:0/3378325361 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195400ce70 con 0x7f195c141910 2023-10-30T12:32:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1943fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1943fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1943fff640 10 monclient: dump: 2023-10-30T12:32:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:37.919 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:37.919 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.919 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.919 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:37.919 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:37.920 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:37.920 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:37.920 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:37.920 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1943fff640 10 monclient: _finish_auth 0 2023-10-30T12:32:37.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1943fff640 10 monclient: _check_auth_tickets 2023-10-30T12:32:37.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1943fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:37.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1943fff640 1 -- 172.21.15.3:0/3378325361 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1954014440 con 0x7f195c141910 2023-10-30T12:32:37.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1943fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:37.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1943fff640 1 -- 172.21.15.3:0/3378325361 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954014820 con 0x7f195c141910 2023-10-30T12:32:37.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1962234640 5 monclient: authenticate success, global_id 21880 2023-10-30T12:32:37.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1943fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:37.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1943fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:37.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1943fff640 10 monclient: dump: 2023-10-30T12:32:37.922 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:37.923 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:37.923 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.923 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:37.923 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:37.923 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:37.924 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:37.924 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:37.924 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:37.924 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:37.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1962234640 10 monclient: _renew_subs 2023-10-30T12:32:37.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1962234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:37.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1962234640 1 -- 172.21.15.3:0/3378325361 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f195c1d7790 con 0x7f195c141910 2023-10-30T12:32:37.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1959ffb640 4 set_mon_vals no callback set 2023-10-30T12:32:37.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1962234640 10 client.21880.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:37.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1962234640 10 monclient: _renew_subs 2023-10-30T12:32:37.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1962234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:37.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1962234640 1 -- 172.21.15.3:0/3378325361 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195c0b5480 con 0x7f195c141910 2023-10-30T12:32:37.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.898+0000 7f1962234640 1 librados: init done 2023-10-30T12:32:37.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.899+0000 7f1962234640 10 client.21880.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:37.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.899+0000 7f1962234640 15 client.21880.objecter _session_command_op_assign -1 1 2023-10-30T12:32:37.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.899+0000 7f1962234640 20 client.21880.objecter _get_session osd=-1 returning homeless 2023-10-30T12:32:37.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.899+0000 7f1962234640 10 client.21880.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:37.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.899+0000 7f1962234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:37.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.899+0000 7f1962234640 1 -- 172.21.15.3:0/3378325361 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f195c1d7a50 con 0x7f195c141910 2023-10-30T12:32:37.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.917+0000 7f1943fff640 1 -- 172.21.15.3:0/3378325361 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f195401d3b0 con 0x7f195c141910 2023-10-30T12:32:37.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.917+0000 7f1943fff640 1 --2- 172.21.15.3:0/3378325361 >> v2:172.21.15.92:6800/1750155735 conn(0x7f194806c7c0 0x7f194806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.918+0000 7f195b7fe640 1 --2- 172.21.15.3:0/3378325361 >> v2:172.21.15.92:6800/1750155735 conn(0x7f194806c7c0 0x7f194806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.918+0000 7f195b7fe640 10 monclient: get_auth_request con 0x7f194806c7c0 auth_method 0 2023-10-30T12:32:37.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.918+0000 7f1943fff640 10 client.21880.objecter ms_handle_connect 0x7f194806c7c0 2023-10-30T12:32:37.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.918+0000 7f195b7fe640 1 --2- 172.21.15.3:0/3378325361 >> v2:172.21.15.92:6800/1750155735 conn(0x7f194806c7c0 0x7f194806ec80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f194c009c00 tx=0x7f194c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 1 -- 172.21.15.3:0/3378325361 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f1954058790 con 0x7f195c141910 2023-10-30T12:32:37.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 10 client.21880.objecter ms_dispatch 0x7f195c000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:37.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 3 client.21880.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:37.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 3 client.21880.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:37.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 10 client.21880.objecter checking command 1 2023-10-30T12:32:37.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 1 --2- 172.21.15.3:0/3378325361 >> v2:172.21.15.92:6809/3275842310 conn(0x7f1948073a30 0x7f1948075ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 20 client.21880.objecter _get_session s=0x7f1954014820 osd=6 3 2023-10-30T12:32:37.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 20 client.21880.objecter put_session s=0x7f1954014820 osd=6 3 2023-10-30T12:32:37.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 15 client.21880.objecter _session_command_op_remove -1 1 2023-10-30T12:32:37.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 20 client.21880.objecter _get_session s=0x7f1954014820 osd=6 3 2023-10-30T12:32:37.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 20 client.21880.objecter get_session s=0x7f1954014820 osd=6 3 2023-10-30T12:32:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 15 client.21880.objecter _session_command_op_assign 6 1 2023-10-30T12:32:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 20 client.21880.objecter put_session s=0x7f1954014820 osd=6 4 2023-10-30T12:32:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 10 client.21880.objecter _send_command 1 2023-10-30T12:32:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 1 -- 172.21.15.3:0/3378325361 --> v2:172.21.15.92:6809/3275842310 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f195400ce70 con 0x7f1948073a30 2023-10-30T12:32:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 20 client.21880.objecter dump_active .. 0 homeless 2023-10-30T12:32:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 1 -- 172.21.15.3:0/3378325361 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1954054ad0 con 0x7f195c141910 2023-10-30T12:32:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.921+0000 7f1943fff640 10 monclient: handle_get_version_reply finishing 1 version 900 2023-10-30T12:32:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.922+0000 7f195affd640 1 --2- 172.21.15.3:0/3378325361 >> v2:172.21.15.92:6809/3275842310 conn(0x7f1948073a30 0x7f1948075ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.922+0000 7f195affd640 10 monclient: get_auth_request con 0x7f1948073a30 auth_method 0 2023-10-30T12:32:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.926+0000 7f195affd640 1 --2- 172.21.15.3:0/3378325361 >> v2:172.21.15.92:6809/3275842310 conn(0x7f1948073a30 0x7f1948075ed0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.926+0000 7f1943fff640 10 client.21880.objecter ms_handle_connect 0x7f1948073a30 2023-10-30T12:32:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.949+0000 7f1943fff640 1 -- 172.21.15.3:0/3378325361 <== osd.6 v2:172.21.15.92:6809/3275842310 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f195400ce70 con 0x7f1948073a30 2023-10-30T12:32:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.949+0000 7f1943fff640 10 client.21880.objecter ms_dispatch 0x7f195c000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:32:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.949+0000 7f1943fff640 10 client.21880.objecter _finish_command 1 = system:0 2023-10-30T12:32:37.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.949+0000 7f1943fff640 20 client.21880.objecter put_session s=0x7f1954014820 osd=6 4 2023-10-30T12:32:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.949+0000 7f1943fff640 15 client.21880.objecter _session_command_op_remove 6 1 2023-10-30T12:32:37.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.965+0000 7f1962234640 10 client.21880.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:32:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.965+0000 7f1962234640 15 client.21880.objecter _session_command_op_assign -1 2 2023-10-30T12:32:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.965+0000 7f1962234640 20 client.21880.objecter _get_session s=0x7f1954014820 osd=6 3 2023-10-30T12:32:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.965+0000 7f1962234640 20 client.21880.objecter put_session s=0x7f1954014820 osd=6 3 2023-10-30T12:32:37.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.965+0000 7f1962234640 20 client.21880.objecter _get_session s=0x7f1954014820 osd=6 3 2023-10-30T12:32:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.965+0000 7f1962234640 15 client.21880.objecter _session_command_op_remove -1 2 2023-10-30T12:32:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.965+0000 7f1962234640 20 client.21880.objecter get_session s=0x7f1954014820 osd=6 3 2023-10-30T12:32:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.965+0000 7f1962234640 15 client.21880.objecter _session_command_op_assign 6 2 2023-10-30T12:32:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.965+0000 7f1962234640 20 client.21880.objecter put_session s=0x7f1954014820 osd=6 4 2023-10-30T12:32:37.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.965+0000 7f1962234640 10 client.21880.objecter _send_command 2 2023-10-30T12:32:37.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.965+0000 7f1962234640 1 -- 172.21.15.3:0/3378325361 --> v2:172.21.15.92:6809/3275842310 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f195c13f4d0 con 0x7f1948073a30 2023-10-30T12:32:37.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1943fff640 1 -- 172.21.15.3:0/3378325361 <== osd.6 v2:172.21.15.92:6809/3275842310 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1950031740 con 0x7f1948073a30 2023-10-30T12:32:37.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1943fff640 10 client.21880.objecter ms_dispatch 0x7f195c000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:32:37.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1943fff640 10 client.21880.objecter _finish_command 2 = system:0 2023-10-30T12:32:37.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1943fff640 20 client.21880.objecter put_session s=0x7f1954014820 osd=6 4 2023-10-30T12:32:37.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1943fff640 15 client.21880.objecter _session_command_op_remove 6 2 2023-10-30T12:32:37.983 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-10-30T12:32:37.983 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:32:37.983 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-10-30T12:32:37.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 10 librados: watch_flush enter 2023-10-30T12:32:37.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 10 librados: watch_flush exit 2023-10-30T12:32:37.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 10 client.21880.objecter close_session for osd.6 2023-10-30T12:32:37.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 1 -- 172.21.15.3:0/3378325361 >> v2:172.21.15.92:6809/3275842310 conn(0x7f1948073a30 msgr2=0x7f1948075ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 1 --2- 172.21.15.3:0/3378325361 >> v2:172.21.15.92:6809/3275842310 conn(0x7f1948073a30 0x7f1948075ed0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 20 client.21880.objecter put_session s=0x7f1954014820 osd=6 1 2023-10-30T12:32:37.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 20 client.21880.objecter shutdown clearing up homeless session... 2023-10-30T12:32:37.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 10 client.21880.objecter successfully canceled tick 2023-10-30T12:32:37.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 1 -- 172.21.15.3:0/3378325361 >> v2:172.21.15.92:6800/1750155735 conn(0x7f194806c7c0 msgr2=0x7f194806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 1 --2- 172.21.15.3:0/3378325361 >> v2:172.21.15.92:6800/1750155735 conn(0x7f194806c7c0 0x7f194806ec80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f194c009c00 tx=0x7f194c009190 comp rx=0 tx=0).stop 2023-10-30T12:32:37.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 10 monclient: shutdown 2023-10-30T12:32:37.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:37.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 1 -- 172.21.15.3:0/3378325361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f195c141910 msgr2=0x7f195c1d7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 1 --2- 172.21.15.3:0/3378325361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f195c141910 0x7f195c1d7250 secure :-1 s=READY pgs=5319 cs=0 l=1 rev1=1 crypto rx=0x7f195400be20 tx=0x7f195400bf10 comp rx=0 tx=0).stop 2023-10-30T12:32:37.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f195bfff640 1 -- 172.21.15.3:0/3378325361 reap_dead start 2023-10-30T12:32:37.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 1 -- 172.21.15.3:0/3378325361 shutdown_connections 2023-10-30T12:32:37.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 1 -- 172.21.15.3:0/3378325361 >> 172.21.15.3:0/3378325361 conn(0x7f195c065ea0 msgr2=0x7f195c1cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:37.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 1 -- 172.21.15.3:0/3378325361 shutdown_connections 2023-10-30T12:32:37.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 1 -- 172.21.15.3:0/3378325361 wait complete. 2023-10-30T12:32:37.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:37.980+0000 7f1962234640 1 librados: shutdown 2023-10-30T12:32:37.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-30T12:32:38.012 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:37.947 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:38.017 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:37.953 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:38.035 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:37.970 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:38.065 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:37.974 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:38.070 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:38.005 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:38.071 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:37.980 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:38.073 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:38.009 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:38.083 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:37.992 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:38.089 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:38.025 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:38.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.104+0000 7f35681dc640 10 monclient: get_monmap_and_config 2023-10-30T12:32:38.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.104+0000 7f35681dc640 10 monclient: build_initial_monmap 2023-10-30T12:32:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.104+0000 7f35681dc640 10 monclient: monmap: 2023-10-30T12:32:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:38.105198+0000 2023-10-30T12:32:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:38.105198+0000 2023-10-30T12:32:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:38.108 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:38.108 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:38.108 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.104+0000 7f35681dc640 1 Processor -- start 2023-10-30T12:32:38.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 1 -- start start 2023-10-30T12:32:38.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 10 monclient: init 2023-10-30T12:32:38.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:38.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2023-10-30T12:32:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f35600b4350 0x7f35600a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 10 monclient(hunting): picked mon.noname-a con 0x7f35600b4350 addr v2:172.21.15.3:3300/0 2023-10-30T12:32:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f356013eb50 0x7f356013efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 10 monclient(hunting): picked mon.noname-c con 0x7f356013eb50 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3560144f50 addr v1:172.21.15.92:6789/0 2023-10-30T12:32:38.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35600aa6e0 con 0x7f3560144f50 2023-10-30T12:32:38.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:38.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35600679f0 con 0x7f35600b4350 2023-10-30T12:32:38.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:38.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35600a8910 con 0x7f356013eb50 2023-10-30T12:32:38.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:38.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.105+0000 7f35681dc640 10 monclient(hunting): authenticate will time out at 2749.319719s 2023-10-30T12:32:38.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.109+0000 7f3565777640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f356013eb50 0x7f356013efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:38.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.110+0000 7f3565f78640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f35600b4350 0x7f35600a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:38.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.111+0000 7f3565777640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f356013eb50 0x7f356013efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44800/0 (socket says 172.21.15.3:44800) 2023-10-30T12:32:38.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.111+0000 7f3565777640 1 -- 172.21.15.3:0/3411191990 learned_addr learned my addr 172.21.15.3:0/3411191990 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:38.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.111+0000 7f3565777640 10 monclient(hunting): get_auth_request con 0x7f356013eb50 auth_method 0 2023-10-30T12:32:38.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.111+0000 7f3565777640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:38.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.111+0000 7f3565777640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:38.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.111+0000 7f3565777640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:38.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.111+0000 7f3565f78640 10 monclient(hunting): get_auth_request con 0x7f35600b4350 auth_method 0 2023-10-30T12:32:38.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.111+0000 7f3565f78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:38.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.111+0000 7f3565f78640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:38.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.111+0000 7f3565f78640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:38.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.113+0000 7f3565f78640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:38.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.113+0000 7f3565f78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:38.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.113+0000 7f3565f78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:38.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.116+0000 7f3565f78640 10 monclient(hunting): handle_auth_done global_id 16689 payload 274 2023-10-30T12:32:38.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.116+0000 7f3565f78640 1 -- 172.21.15.3:0/3411191990 >> v2:172.21.15.92:3300/0 conn(0x7f356013eb50 msgr2=0x7f356013efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:38.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.116+0000 7f3565f78640 1 --2- 172.21.15.3:0/3411191990 >> v2:172.21.15.92:3300/0 conn(0x7f356013eb50 0x7f356013efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:38.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.116+0000 7f3565f78640 1 -- 172.21.15.3:0/3411191990 >> v1:172.21.15.92:6789/0 conn(0x7f3560144f50 legacy=0x7f35601453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:38.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.116+0000 7f3565f78640 10 monclient: _finish_hunting 0 2023-10-30T12:32:38.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.116+0000 7f3565f78640 1 monclient: found mon.noname-a 2023-10-30T12:32:38.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.116+0000 7f3565f78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:38.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.116+0000 7f3565f78640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:32:38.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.116+0000 7f3565f78640 1 -- 172.21.15.3:0/3411191990 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35600a8bc0 con 0x7f35600b4350 2023-10-30T12:32:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.119+0000 7f3565f78640 1 --2- 172.21.15.3:0/3411191990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35600b4350 0x7f35600a83d0 secure :-1 s=READY pgs=3615 cs=0 l=1 rev1=1 crypto rx=0x7f35600aaa90 tx=0x7f355c02b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3fe06a5a4ec382 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:38.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f3564f76640 1 -- 172.21.15.3:0/3411191990 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c02e050 con 0x7f35600b4350 2023-10-30T12:32:38.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f3564f76640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:38.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f3564f76640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:32:38.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f3564f76640 10 monclient: dump: 2023-10-30T12:32:38.132 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:38.132 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:38.132 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.133 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.133 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:38.133 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:38.133 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:38.133 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:38.134 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:38.134 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:38.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f3564f76640 10 monclient: _finish_auth 0 2023-10-30T12:32:38.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f3564f76640 10 monclient: _check_auth_tickets 2023-10-30T12:32:38.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f3564f76640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:38.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f3564f76640 1 -- 172.21.15.3:0/3411191990 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c030710 con 0x7f35600b4350 2023-10-30T12:32:38.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f3564f76640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:38.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f3564f76640 1 -- 172.21.15.3:0/3411191990 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c030ab0 con 0x7f35600b4350 2023-10-30T12:32:38.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f3564f76640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:38.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f3564f76640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:32:38.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f3564f76640 10 monclient: dump: 2023-10-30T12:32:38.136 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:38.136 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:38.136 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.136 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:38.137 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:38.138 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:38.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f35681dc640 5 monclient: authenticate success, global_id 16689 2023-10-30T12:32:38.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f35681dc640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:38.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f35681dc640 4 set_mon_vals no callback set 2023-10-30T12:32:38.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f35681dc640 10 monclient: shutdown 2023-10-30T12:32:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f35681dc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f35681dc640 1 -- 172.21.15.3:0/3411191990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35600b4350 msgr2=0x7f35600a83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.130+0000 7f35681dc640 1 --2- 172.21.15.3:0/3411191990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35600b4350 0x7f35600a83d0 secure :-1 s=READY pgs=3615 cs=0 l=1 rev1=1 crypto rx=0x7f35600aaa90 tx=0x7f355c02b730 comp rx=0 tx=0).stop 2023-10-30T12:32:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.131+0000 7f35681dc640 1 -- 172.21.15.3:0/3411191990 shutdown_connections 2023-10-30T12:32:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.131+0000 7f35681dc640 1 --2- 172.21.15.3:0/3411191990 >> v2:172.21.15.92:3300/0 conn(0x7f356013eb50 0x7f356013efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.131+0000 7f35681dc640 1 --2- 172.21.15.3:0/3411191990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35600b4350 0x7f35600a83d0 unknown :-1 s=CLOSED pgs=3615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.131+0000 7f35681dc640 1 -- 172.21.15.3:0/3411191990 >> 172.21.15.3:0/3411191990 conn(0x7f3560065ea0 msgr2=0x7f35600662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.131+0000 7f35681dc640 1 -- 172.21.15.3:0/3411191990 shutdown_connections 2023-10-30T12:32:38.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.131+0000 7f35681dc640 1 -- 172.21.15.3:0/3411191990 wait complete. 2023-10-30T12:32:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.132+0000 7f35681dc640 10 monclient: build_initial_monmap 2023-10-30T12:32:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.132+0000 7f35681dc640 10 monclient: monmap: 2023-10-30T12:32:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:38.143 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:38.144 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:38.144 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:38.144 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:38.144 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:38.144 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:38.145 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:38.145 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:38.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.132+0000 7f35681dc640 1 librados: starting msgr at 2023-10-30T12:32:38.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.132+0000 7f35681dc640 1 librados: starting objecter 2023-10-30T12:32:38.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.132+0000 7f35681dc640 1 Processor -- start 2023-10-30T12:32:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.132+0000 7f35681dc640 1 -- start start 2023-10-30T12:32:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.132+0000 7f35681dc640 1 librados: setting wanted keys 2023-10-30T12:32:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.132+0000 7f35681dc640 1 librados: calling monclient init 2023-10-30T12:32:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.132+0000 7f35681dc640 10 monclient: init 2023-10-30T12:32:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:32:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f35600b4350 0x7f35600b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 10 monclient(hunting): picked mon.noname-c con 0x7f35600b4350 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f356013eb50 0x7f3560141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 10 monclient(hunting): picked mon.noname-b con 0x7f356013eb50 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3560144f50 0x7f35601416e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 10 monclient(hunting): picked mon.noname-a con 0x7f3560144f50 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35600b2d60 con 0x7f3560144f50 2023-10-30T12:32:38.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3560142380 con 0x7f356013eb50 2023-10-30T12:32:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35601d9df0 con 0x7f35600b4350 2023-10-30T12:32:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.133+0000 7f35681dc640 10 monclient(hunting): authenticate will time out at 2749.347212s 2023-10-30T12:32:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.137+0000 7f3566779640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3560144f50 0x7f35601416e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.139+0000 7f3566779640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3560144f50 0x7f35601416e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40576/0 (socket says 172.21.15.3:40576) 2023-10-30T12:32:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.139+0000 7f3566779640 1 -- 172.21.15.3:0/2923601467 learned_addr learned my addr 172.21.15.3:0/2923601467 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.139+0000 7f3566779640 10 monclient(hunting): get_auth_request con 0x7f3560144f50 auth_method 0 2023-10-30T12:32:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.139+0000 7f3566779640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.139+0000 7f3566779640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.139+0000 7f3566779640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.140+0000 7f3566779640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.140+0000 7f3566779640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.140+0000 7f3566779640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.146+0000 7f3566779640 10 monclient(hunting): handle_auth_done global_id 16692 payload 824 2023-10-30T12:32:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.146+0000 7f3566779640 1 -- 172.21.15.3:0/2923601467 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f35600b4350 msgr2=0x7f35600b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.146+0000 7f3566779640 1 --2- 172.21.15.3:0/2923601467 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f35600b4350 0x7f35600b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.146+0000 7f3566779640 1 -- 172.21.15.3:0/2923601467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f356013eb50 msgr2=0x7f3560141cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.146+0000 7f3566779640 1 --2- 172.21.15.3:0/2923601467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f356013eb50 0x7f3560141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.146+0000 7f3566779640 10 monclient: _finish_hunting 0 2023-10-30T12:32:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.146+0000 7f3566779640 1 monclient: found mon.noname-a 2023-10-30T12:32:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.146+0000 7f3566779640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.146+0000 7f3566779640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:32:38.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.146+0000 7f3566779640 1 -- 172.21.15.3:0/2923601467 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35601d9fc0 con 0x7f3560144f50 2023-10-30T12:32:38.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.155+0000 7f3566779640 1 --2- 172.21.15.3:0/2923601467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3560144f50 0x7f35601416e0 secure :-1 s=READY pgs=3616 cs=0 l=1 rev1=1 crypto rx=0x7f355800b990 tx=0x7f355800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:38.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.155+0000 7f35567fc640 10 client.?.objecter ms_handle_connect 0x7f3560144f50 2023-10-30T12:32:38.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.155+0000 7f35567fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:38.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35567fc640 1 -- 172.21.15.3:0/2923601467 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355800cba0 con 0x7f3560144f50 2023-10-30T12:32:38.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:38.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:32:38.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35567fc640 10 monclient: dump: 2023-10-30T12:32:38.232 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:38.233 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:38.233 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.233 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.233 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:38.233 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:38.234 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:38.234 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:38.234 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:38.234 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:38.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35567fc640 10 monclient: _finish_auth 0 2023-10-30T12:32:38.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35567fc640 10 monclient: _check_auth_tickets 2023-10-30T12:32:38.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:38.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35567fc640 1 -- 172.21.15.3:0/2923601467 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355800cd10 con 0x7f3560144f50 2023-10-30T12:32:38.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35681dc640 5 monclient: authenticate success, global_id 16692 2023-10-30T12:32:38.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35681dc640 10 monclient: _renew_subs 2023-10-30T12:32:38.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35681dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:32:38.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35681dc640 1 -- 172.21.15.3:0/2923601467 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35601da220 con 0x7f3560144f50 2023-10-30T12:32:38.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35567fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:38.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35567fc640 1 -- 172.21.15.3:0/2923601467 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355800ce90 con 0x7f3560144f50 2023-10-30T12:32:38.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:38.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:32:38.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35567fc640 10 monclient: dump: 2023-10-30T12:32:38.237 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:38.237 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:38.238 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.238 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.238 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:38.238 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:38.238 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:38.239 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:38.239 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:38.239 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:38.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f3557fff640 4 set_mon_vals no callback set 2023-10-30T12:32:38.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35681dc640 10 client.16692.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:38.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35681dc640 10 monclient: _renew_subs 2023-10-30T12:32:38.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35681dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:32:38.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35681dc640 1 -- 172.21.15.3:0/2923601467 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35601da790 con 0x7f3560144f50 2023-10-30T12:32:38.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.229+0000 7f35681dc640 1 librados: init done 2023-10-30T12:32:38.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.230+0000 7f35681dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:38.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.230+0000 7f35681dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:38.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.230+0000 7f35681dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.3:3300/0 2023-10-30T12:32:38.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.230+0000 7f35681dc640 1 -- 172.21.15.3:0/2923601467 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356004e9b0 con 0x7f3560144f50 2023-10-30T12:32:38.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.504+0000 7f35567fc640 1 -- 172.21.15.3:0/2923601467 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f355801d3b0 con 0x7f3560144f50 2023-10-30T12:32:38.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.505+0000 7f35567fc640 1 --2- 172.21.15.3:0/2923601467 >> v2:172.21.15.92:6800/1750155735 conn(0x7f354406c7c0 0x7f354406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:38.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.505+0000 7f3565f78640 1 --2- 172.21.15.3:0/2923601467 >> v2:172.21.15.92:6800/1750155735 conn(0x7f354406c7c0 0x7f354406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:38.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.505+0000 7f3565f78640 10 monclient: get_auth_request con 0x7f354406c7c0 auth_method 0 2023-10-30T12:32:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.506+0000 7f3565f78640 1 --2- 172.21.15.3:0/2923601467 >> v2:172.21.15.92:6800/1750155735 conn(0x7f354406c7c0 0x7f354406ec80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f355c034000 tx=0x7f355c02f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:38.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.506+0000 7f35567fc640 10 client.16692.objecter ms_handle_connect 0x7f354406c7c0 2023-10-30T12:32:38.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.520+0000 7f35567fc640 1 -- 172.21.15.3:0/2923601467 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f35580589b0 con 0x7f3560144f50 2023-10-30T12:32:38.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.520+0000 7f35567fc640 10 client.16692.objecter ms_dispatch 0x7f3560000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:38.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.520+0000 7f35567fc640 3 client.16692.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:38.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.520+0000 7f35567fc640 3 client.16692.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:38.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.520+0000 7f35567fc640 20 client.16692.objecter dump_active .. 0 homeless 2023-10-30T12:32:38.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.520+0000 7f35567fc640 1 -- 172.21.15.3:0/2923601467 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f35580582b0 con 0x7f3560144f50 2023-10-30T12:32:38.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.520+0000 7f35567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:38.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.520+0000 7f35567fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:32:38.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.738+0000 7f35681dc640 1 -- 172.21.15.3:0/2923601467 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f35601da790 con 0x7f354406c7c0 2023-10-30T12:32:38.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.739+0000 7f35567fc640 1 -- 172.21.15.3:0/2923601467 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f35601da790 con 0x7f354406c7c0 2023-10-30T12:32:38.740 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 6 to deep-scrub 2023-10-30T12:32:38.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 10 librados: watch_flush enter 2023-10-30T12:32:38.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 10 librados: watch_flush exit 2023-10-30T12:32:38.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 20 client.16692.objecter shutdown clearing up homeless session... 2023-10-30T12:32:38.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 10 client.16692.objecter successfully canceled tick 2023-10-30T12:32:38.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 1 -- 172.21.15.3:0/2923601467 >> v2:172.21.15.92:6800/1750155735 conn(0x7f354406c7c0 msgr2=0x7f354406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:38.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 1 --2- 172.21.15.3:0/2923601467 >> v2:172.21.15.92:6800/1750155735 conn(0x7f354406c7c0 0x7f354406ec80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f355c034000 tx=0x7f355c02f000 comp rx=0 tx=0).stop 2023-10-30T12:32:38.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 10 monclient: shutdown 2023-10-30T12:32:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 1 -- 172.21.15.3:0/2923601467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3560144f50 msgr2=0x7f35601416e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 1 --2- 172.21.15.3:0/2923601467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3560144f50 0x7f35601416e0 secure :-1 s=READY pgs=3616 cs=0 l=1 rev1=1 crypto rx=0x7f355800b990 tx=0x7f355800be50 comp rx=0 tx=0).stop 2023-10-30T12:32:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 1 -- 172.21.15.3:0/2923601467 shutdown_connections 2023-10-30T12:32:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 1 --2- 172.21.15.3:0/2923601467 >> v2:172.21.15.92:6800/1750155735 conn(0x7f354406c7c0 0x7f354406ec80 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 1 --2- 172.21.15.3:0/2923601467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3560144f50 0x7f35601416e0 unknown :-1 s=CLOSED pgs=3616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 1 --2- 172.21.15.3:0/2923601467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f356013eb50 0x7f3560141cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 1 --2- 172.21.15.3:0/2923601467 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f35600b4350 0x7f35600b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 1 -- 172.21.15.3:0/2923601467 >> 172.21.15.3:0/2923601467 conn(0x7f3560065ea0 msgr2=0x7f356013c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 1 -- 172.21.15.3:0/2923601467 shutdown_connections 2023-10-30T12:32:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 1 -- 172.21.15.3:0/2923601467 wait complete. 2023-10-30T12:32:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.741+0000 7f35681dc640 1 librados: shutdown 2023-10-30T12:32:38.755 INFO:tasks.ceph:Scrubbing osd.7 2023-10-30T12:32:38.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:32:38.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.905+0000 7f1ed5f47640 10 monclient: get_monmap_and_config 2023-10-30T12:32:38.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.905+0000 7f1ed5f47640 10 monclient: build_initial_monmap 2023-10-30T12:32:38.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.905+0000 7f1ed5f47640 10 monclient: monmap: 2023-10-30T12:32:38.907 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:38.908 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:38.908 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:38.906555+0000 2023-10-30T12:32:38.908 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:38.906555+0000 2023-10-30T12:32:38.908 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:38.908 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:38.909 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:38.909 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:38.909 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:38.909 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:38.909 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:38.910 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:38.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.906+0000 7f1ed5f47640 1 Processor -- start 2023-10-30T12:32:38.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.906+0000 7f1ed5f47640 1 -- start start 2023-10-30T12:32:38.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.906+0000 7f1ed5f47640 10 monclient: init 2023-10-30T12:32:38.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.906+0000 7f1ed5f47640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:38.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.906+0000 7f1ed5f47640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-10-30T12:32:38.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.906+0000 7f1ed5f47640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1ed00b4360 0x7f1ed013ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:38.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.906+0000 7f1ed5f47640 10 monclient(hunting): picked mon.noname-a con 0x7f1ed00b4360 addr v2:172.21.15.3:3300/0 2023-10-30T12:32:38.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.906+0000 7f1ed5f47640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1ed00b12b0 0x7f1ed013f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:38.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.906+0000 7f1ed5f47640 10 monclient(hunting): picked mon.noname-c con 0x7f1ed00b12b0 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:38.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.906+0000 7f1ed5f47640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ed0144e10 addr v1:172.21.15.3:6789/0 2023-10-30T12:32:38.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.906+0000 7f1ed5f47640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ed00aa180 con 0x7f1ed0144e10 2023-10-30T12:32:38.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.906+0000 7f1ed5f47640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:38.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.906+0000 7f1ed5f47640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ed0067a30 con 0x7f1ed00b4360 2023-10-30T12:32:38.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.906+0000 7f1ed5f47640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:38.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.906+0000 7f1ed5f47640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ed00b4740 con 0x7f1ed00b12b0 2023-10-30T12:32:38.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.907+0000 7f1ed5f47640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:38.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.907+0000 7f1ed5f47640 10 monclient(hunting): authenticate will time out at 2750.121070s 2023-10-30T12:32:38.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.907+0000 7f1eceffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1ed00b12b0 0x7f1ed013f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:38.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.908+0000 7f1eceffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1ed00b12b0 0x7f1ed013f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44818/0 (socket says 172.21.15.3:44818) 2023-10-30T12:32:38.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.908+0000 7f1eceffd640 1 -- 172.21.15.3:0/3601758796 learned_addr learned my addr 172.21.15.3:0/3601758796 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:38.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.908+0000 7f1eceffd640 10 monclient(hunting): get_auth_request con 0x7f1ed00b12b0 auth_method 0 2023-10-30T12:32:38.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.908+0000 7f1eceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:38.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.908+0000 7f1eceffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:38.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.908+0000 7f1eceffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:38.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.908+0000 7f1eceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:38.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.908+0000 7f1eceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:38.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.908+0000 7f1eceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:38.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.910+0000 7f1eceffd640 10 monclient(hunting): handle_auth_done global_id 21886 payload 274 2023-10-30T12:32:38.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.910+0000 7f1eceffd640 1 -- 172.21.15.3:0/3601758796 >> v2:172.21.15.3:3300/0 conn(0x7f1ed00b4360 msgr2=0x7f1ed013ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:38.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.910+0000 7f1eceffd640 1 --2- 172.21.15.3:0/3601758796 >> v2:172.21.15.3:3300/0 conn(0x7f1ed00b4360 0x7f1ed013ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:38.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.910+0000 7f1eceffd640 1 -- 172.21.15.3:0/3601758796 >> v1:172.21.15.3:6789/0 conn(0x7f1ed0144e10 legacy=0x7f1ed0145260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:38.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.910+0000 7f1eceffd640 10 monclient: _finish_hunting 0 2023-10-30T12:32:38.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.910+0000 7f1eceffd640 1 monclient: found mon.noname-c 2023-10-30T12:32:38.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.910+0000 7f1eceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:38.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.910+0000 7f1eceffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:32:38.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.910+0000 7f1eceffd640 1 -- 172.21.15.3:0/3601758796 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed0145970 con 0x7f1ed00b12b0 2023-10-30T12:32:38.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.912+0000 7f1eceffd640 1 --2- 172.21.15.3:0/3601758796 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed00b12b0 0x7f1ed013f150 secure :-1 s=READY pgs=5320 cs=0 l=1 rev1=1 crypto rx=0x7f1ed006be80 tx=0x7f1ec002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d850ac1ccf6805c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:38.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ece7fc640 1 -- 172.21.15.3:0/3601758796 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00317e0 con 0x7f1ed00b12b0 2023-10-30T12:32:38.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ece7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:38.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ece7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:32:38.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ece7fc640 10 monclient: dump: 2023-10-30T12:32:38.922 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:38.922 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:38.922 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.923 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.923 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:38.923 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:38.923 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:38.923 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:38.923 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:38.924 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:38.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ece7fc640 10 monclient: _finish_auth 0 2023-10-30T12:32:38.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ece7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:32:38.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ece7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:38.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ece7fc640 1 -- 172.21.15.3:0/3601758796 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec0031950 con 0x7f1ed00b12b0 2023-10-30T12:32:38.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ece7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:38.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ece7fc640 1 -- 172.21.15.3:0/3601758796 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0031cf0 con 0x7f1ed00b12b0 2023-10-30T12:32:38.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ece7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:38.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ece7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:38.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ece7fc640 10 monclient: dump: 2023-10-30T12:32:38.926 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:38.926 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:38.926 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.926 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.926 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:38.927 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:38.927 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:38.927 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:38.927 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:38.927 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:38.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ed5f47640 5 monclient: authenticate success, global_id 21886 2023-10-30T12:32:38.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ed5f47640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:38.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.915+0000 7f1ed5f47640 4 set_mon_vals no callback set 2023-10-30T12:32:38.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.916+0000 7f1ed5f47640 10 monclient: shutdown 2023-10-30T12:32:38.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.916+0000 7f1ed5f47640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:38.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.916+0000 7f1ed5f47640 1 -- 172.21.15.3:0/3601758796 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed00b12b0 msgr2=0x7f1ed013f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:38.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.916+0000 7f1ed5f47640 1 --2- 172.21.15.3:0/3601758796 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed00b12b0 0x7f1ed013f150 secure :-1 s=READY pgs=5320 cs=0 l=1 rev1=1 crypto rx=0x7f1ed006be80 tx=0x7f1ec002b880 comp rx=0 tx=0).stop 2023-10-30T12:32:38.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.916+0000 7f1ed5f47640 1 -- 172.21.15.3:0/3601758796 shutdown_connections 2023-10-30T12:32:38.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.916+0000 7f1ed5f47640 1 --2- 172.21.15.3:0/3601758796 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed00b12b0 0x7f1ed013f150 unknown :-1 s=CLOSED pgs=5320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:38.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.916+0000 7f1ed5f47640 1 --2- 172.21.15.3:0/3601758796 >> v2:172.21.15.3:3300/0 conn(0x7f1ed00b4360 0x7f1ed013ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:38.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.916+0000 7f1ed5f47640 1 -- 172.21.15.3:0/3601758796 >> 172.21.15.3:0/3601758796 conn(0x7f1ed0065ed0 msgr2=0x7f1ed00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:38.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.916+0000 7f1ed5f47640 1 -- 172.21.15.3:0/3601758796 shutdown_connections 2023-10-30T12:32:38.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.916+0000 7f1ed5f47640 1 -- 172.21.15.3:0/3601758796 wait complete. 2023-10-30T12:32:38.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.917+0000 7f1ed5f47640 10 monclient: build_initial_monmap 2023-10-30T12:32:38.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.917+0000 7f1ed5f47640 10 monclient: monmap: 2023-10-30T12:32:38.931 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:38.931 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:38.931 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:38.931 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:38.931 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:38.931 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:38.932 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:38.932 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:38.932 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:38.932 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:38.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.917+0000 7f1ed5f47640 1 librados: starting msgr at 2023-10-30T12:32:38.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.917+0000 7f1ed5f47640 1 librados: starting objecter 2023-10-30T12:32:38.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.917+0000 7f1ed5f47640 1 Processor -- start 2023-10-30T12:32:38.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.917+0000 7f1ed5f47640 1 -- start start 2023-10-30T12:32:38.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.917+0000 7f1ed5f47640 1 librados: setting wanted keys 2023-10-30T12:32:38.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.917+0000 7f1ed5f47640 1 librados: calling monclient init 2023-10-30T12:32:38.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.917+0000 7f1ed5f47640 10 monclient: init 2023-10-30T12:32:38.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:38.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:32:38.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed00b12b0 0x7f1ed0143ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:38.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 10 monclient(hunting): picked mon.noname-b con 0x7f1ed00b12b0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:38.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1ed00b4360 0x7f1ed013ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:38.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 10 monclient(hunting): picked mon.noname-c con 0x7f1ed00b4360 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:38.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed0144e10 0x7f1ed013b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:38.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 10 monclient(hunting): picked mon.noname-a con 0x7f1ed0144e10 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:38.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:38.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed0145470 con 0x7f1ed0144e10 2023-10-30T12:32:38.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:38.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed013bb20 con 0x7f1ed00b12b0 2023-10-30T12:32:38.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:38.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed013bfb0 con 0x7f1ed00b4360 2023-10-30T12:32:38.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:38.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1ed5f47640 10 monclient(hunting): authenticate will time out at 2750.132421s 2023-10-30T12:32:38.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.918+0000 7f1eceffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1ed00b4360 0x7f1ed013ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:38.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1ecf7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed00b12b0 0x7f1ed0143ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:38.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1eceffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1ed00b4360 0x7f1ed013ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:37750/0 (socket says 172.21.15.3:37750) 2023-10-30T12:32:38.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1eceffd640 1 -- 172.21.15.3:0/143432934 learned_addr learned my addr 172.21.15.3:0/143432934 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:38.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1eceffd640 10 monclient(hunting): get_auth_request con 0x7f1ed00b4360 auth_method 0 2023-10-30T12:32:38.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1eceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:38.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1eceffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:38.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1eceffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:38.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1ecf7fe640 10 monclient(hunting): get_auth_request con 0x7f1ed00b12b0 auth_method 0 2023-10-30T12:32:38.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1ecf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:38.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1ecf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:38.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1ecf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:38.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1ecf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:38.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1ecf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:38.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1ecf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:38.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1eceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:38.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1eceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:38.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.919+0000 7f1eceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:38.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.922+0000 7f1ecf7fe640 10 monclient(hunting): handle_auth_done global_id 21889 payload 824 2023-10-30T12:32:38.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.922+0000 7f1ecf7fe640 1 -- 172.21.15.3:0/143432934 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1ed00b4360 msgr2=0x7f1ed013ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:38.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.922+0000 7f1ecf7fe640 1 --2- 172.21.15.3:0/143432934 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1ed00b4360 0x7f1ed013ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:38.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.922+0000 7f1ecf7fe640 1 -- 172.21.15.3:0/143432934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed0144e10 msgr2=0x7f1ed013b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:38.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.922+0000 7f1ecf7fe640 1 --2- 172.21.15.3:0/143432934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed0144e10 0x7f1ed013b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:38.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.922+0000 7f1ecf7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:32:38.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.922+0000 7f1ecf7fe640 1 monclient: found mon.noname-b 2023-10-30T12:32:38.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.922+0000 7f1ecf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:38.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.922+0000 7f1ecf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:38.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.922+0000 7f1ecf7fe640 1 -- 172.21.15.3:0/143432934 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed01d9ee0 con 0x7f1ed00b12b0 2023-10-30T12:32:38.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.924+0000 7f1ecf7fe640 1 --2- 172.21.15.3:0/143432934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed00b12b0 0x7f1ed0143ed0 secure :-1 s=READY pgs=5321 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400b3f0 tx=0x7f1ec400b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:38.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.924+0000 7f1eb7fff640 10 client.?.objecter ms_handle_connect 0x7f1ed00b12b0 2023-10-30T12:32:38.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.924+0000 7f1eb7fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:38.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.927+0000 7f1eb7fff640 1 -- 172.21.15.3:0/143432934 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec4011830 con 0x7f1ed00b12b0 2023-10-30T12:32:38.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.927+0000 7f1eb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:38.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.927+0000 7f1eb7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:38.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.927+0000 7f1eb7fff640 10 monclient: dump: 2023-10-30T12:32:38.945 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:38.945 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:38.945 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.945 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.946 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:38.946 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:38.946 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:38.946 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:38.946 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:38.947 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:38.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1eb7fff640 10 monclient: _finish_auth 0 2023-10-30T12:32:38.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1eb7fff640 10 monclient: _check_auth_tickets 2023-10-30T12:32:38.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1eb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:38.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1ed5f47640 5 monclient: authenticate success, global_id 21889 2023-10-30T12:32:38.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1eb7fff640 1 -- 172.21.15.3:0/143432934 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec400d040 con 0x7f1ed00b12b0 2023-10-30T12:32:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1eb7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1eb7fff640 1 -- 172.21.15.3:0/143432934 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec4011be0 con 0x7f1ed00b12b0 2023-10-30T12:32:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1eb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1eb7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1eb7fff640 10 monclient: dump: 2023-10-30T12:32:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:38.950 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:38.950 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:38.950 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:38.950 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:38.950 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:38.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1ecdffb640 4 set_mon_vals no callback set 2023-10-30T12:32:38.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1ed5f47640 10 monclient: _renew_subs 2023-10-30T12:32:38.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1ed5f47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:38.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1ed5f47640 1 -- 172.21.15.3:0/143432934 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed01da0e0 con 0x7f1ed00b12b0 2023-10-30T12:32:38.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1ed5f47640 10 client.21889.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:38.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1ed5f47640 10 monclient: _renew_subs 2023-10-30T12:32:38.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1ed5f47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:38.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1ed5f47640 1 -- 172.21.15.3:0/143432934 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed01da580 con 0x7f1ed00b12b0 2023-10-30T12:32:38.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.928+0000 7f1ed5f47640 1 librados: init done 2023-10-30T12:32:38.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.929+0000 7f1ed5f47640 10 client.21889.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:38.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.929+0000 7f1ed5f47640 15 client.21889.objecter _session_command_op_assign -1 1 2023-10-30T12:32:38.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.929+0000 7f1ed5f47640 20 client.21889.objecter _get_session osd=-1 returning homeless 2023-10-30T12:32:38.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.929+0000 7f1ed5f47640 10 client.21889.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:38.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.929+0000 7f1ed5f47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:38.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.929+0000 7f1ed5f47640 1 -- 172.21.15.3:0/143432934 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1ed00aa6b0 con 0x7f1ed00b12b0 2023-10-30T12:32:38.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.949+0000 7f1eb7fff640 1 -- 172.21.15.3:0/143432934 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f1ec4011d50 con 0x7f1ed00b12b0 2023-10-30T12:32:38.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.949+0000 7f1eb7fff640 1 --2- 172.21.15.3:0/143432934 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1ebc06c7c0 0x7f1ebc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:38.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.949+0000 7f1eceffd640 1 --2- 172.21.15.3:0/143432934 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1ebc06c7c0 0x7f1ebc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:38.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.949+0000 7f1eceffd640 10 monclient: get_auth_request con 0x7f1ebc06c7c0 auth_method 0 2023-10-30T12:32:38.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.950+0000 7f1eceffd640 1 --2- 172.21.15.3:0/143432934 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1ebc06c7c0 0x7f1ebc06ec80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1ec002b8c0 tx=0x7f1ec003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:38.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.950+0000 7f1eb7fff640 10 client.21889.objecter ms_handle_connect 0x7f1ebc06c7c0 2023-10-30T12:32:38.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 1 -- 172.21.15.3:0/143432934 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f1ec4058720 con 0x7f1ed00b12b0 2023-10-30T12:32:38.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 10 client.21889.objecter ms_dispatch 0x7f1ed0000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:38.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 3 client.21889.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:38.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 3 client.21889.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:38.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 10 client.21889.objecter checking command 1 2023-10-30T12:32:38.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 1 --2- 172.21.15.3:0/143432934 >> v2:172.21.15.92:6813/1101825516 conn(0x7f1ebc073a30 0x7f1ebc075ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:38.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 20 client.21889.objecter _get_session s=0x7f1ec4011be0 osd=7 3 2023-10-30T12:32:38.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 20 client.21889.objecter put_session s=0x7f1ec4011be0 osd=7 3 2023-10-30T12:32:38.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 15 client.21889.objecter _session_command_op_remove -1 1 2023-10-30T12:32:38.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 20 client.21889.objecter _get_session s=0x7f1ec4011be0 osd=7 3 2023-10-30T12:32:38.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 20 client.21889.objecter get_session s=0x7f1ec4011be0 osd=7 3 2023-10-30T12:32:38.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 15 client.21889.objecter _session_command_op_assign 7 1 2023-10-30T12:32:38.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 20 client.21889.objecter put_session s=0x7f1ec4011be0 osd=7 4 2023-10-30T12:32:38.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 10 client.21889.objecter _send_command 1 2023-10-30T12:32:38.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 1 -- 172.21.15.3:0/143432934 --> v2:172.21.15.92:6813/1101825516 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ec4011830 con 0x7f1ebc073a30 2023-10-30T12:32:38.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 20 client.21889.objecter dump_active .. 0 homeless 2023-10-30T12:32:38.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 1 -- 172.21.15.3:0/143432934 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ec4054360 con 0x7f1ed00b12b0 2023-10-30T12:32:38.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1eb7fff640 10 monclient: handle_get_version_reply finishing 1 version 900 2023-10-30T12:32:38.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.952+0000 7f1ecffff640 1 --2- 172.21.15.3:0/143432934 >> v2:172.21.15.92:6813/1101825516 conn(0x7f1ebc073a30 0x7f1ebc075ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:38.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.953+0000 7f1ecffff640 10 monclient: get_auth_request con 0x7f1ebc073a30 auth_method 0 2023-10-30T12:32:38.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.955+0000 7f1ecffff640 1 --2- 172.21.15.3:0/143432934 >> v2:172.21.15.92:6813/1101825516 conn(0x7f1ebc073a30 0x7f1ebc075ed0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:38.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.955+0000 7f1eb7fff640 10 client.21889.objecter ms_handle_connect 0x7f1ebc073a30 2023-10-30T12:32:38.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.977+0000 7f1eb7fff640 1 -- 172.21.15.3:0/143432934 <== osd.7 v2:172.21.15.92:6813/1101825516 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f1ec4011830 con 0x7f1ebc073a30 2023-10-30T12:32:38.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.977+0000 7f1eb7fff640 10 client.21889.objecter ms_dispatch 0x7f1ed0000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:32:38.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.977+0000 7f1eb7fff640 10 client.21889.objecter _finish_command 1 = system:0 2023-10-30T12:32:38.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.977+0000 7f1eb7fff640 20 client.21889.objecter put_session s=0x7f1ec4011be0 osd=7 4 2023-10-30T12:32:38.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.977+0000 7f1eb7fff640 15 client.21889.objecter _session_command_op_remove 7 1 2023-10-30T12:32:38.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.991+0000 7f1ed5f47640 10 client.21889.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:32:38.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.991+0000 7f1ed5f47640 15 client.21889.objecter _session_command_op_assign -1 2 2023-10-30T12:32:38.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.991+0000 7f1ed5f47640 20 client.21889.objecter _get_session s=0x7f1ec4011be0 osd=7 3 2023-10-30T12:32:38.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.991+0000 7f1ed5f47640 20 client.21889.objecter put_session s=0x7f1ec4011be0 osd=7 3 2023-10-30T12:32:38.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.991+0000 7f1ed5f47640 20 client.21889.objecter _get_session s=0x7f1ec4011be0 osd=7 3 2023-10-30T12:32:38.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.991+0000 7f1ed5f47640 15 client.21889.objecter _session_command_op_remove -1 2 2023-10-30T12:32:38.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.991+0000 7f1ed5f47640 20 client.21889.objecter get_session s=0x7f1ec4011be0 osd=7 3 2023-10-30T12:32:38.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.991+0000 7f1ed5f47640 15 client.21889.objecter _session_command_op_assign 7 2 2023-10-30T12:32:38.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.991+0000 7f1ed5f47640 20 client.21889.objecter put_session s=0x7f1ec4011be0 osd=7 4 2023-10-30T12:32:38.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.991+0000 7f1ed5f47640 10 client.21889.objecter _send_command 2 2023-10-30T12:32:38.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:38.991+0000 7f1ed5f47640 1 -- 172.21.15.3:0/143432934 --> v2:172.21.15.92:6813/1101825516 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1ed00aa3d0 con 0x7f1ebc073a30 2023-10-30T12:32:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1eb7fff640 1 -- 172.21.15.3:0/143432934 <== osd.7 v2:172.21.15.92:6813/1101825516 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1ec800a720 con 0x7f1ebc073a30 2023-10-30T12:32:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1eb7fff640 10 client.21889.objecter ms_dispatch 0x7f1ed0000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:32:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1eb7fff640 10 client.21889.objecter _finish_command 2 = system:0 2023-10-30T12:32:39.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1eb7fff640 20 client.21889.objecter put_session s=0x7f1ec4011be0 osd=7 4 2023-10-30T12:32:39.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1eb7fff640 15 client.21889.objecter _session_command_op_remove 7 2 2023-10-30T12:32:39.008 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-10-30T12:32:39.008 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:32:39.009 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-10-30T12:32:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 10 librados: watch_flush enter 2023-10-30T12:32:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 10 librados: watch_flush exit 2023-10-30T12:32:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 10 client.21889.objecter close_session for osd.7 2023-10-30T12:32:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 1 -- 172.21.15.3:0/143432934 >> v2:172.21.15.92:6813/1101825516 conn(0x7f1ebc073a30 msgr2=0x7f1ebc075ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 1 --2- 172.21.15.3:0/143432934 >> v2:172.21.15.92:6813/1101825516 conn(0x7f1ebc073a30 0x7f1ebc075ed0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 20 client.21889.objecter put_session s=0x7f1ec4011be0 osd=7 1 2023-10-30T12:32:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 20 client.21889.objecter shutdown clearing up homeless session... 2023-10-30T12:32:39.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 10 client.21889.objecter successfully canceled tick 2023-10-30T12:32:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 1 -- 172.21.15.3:0/143432934 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1ebc06c7c0 msgr2=0x7f1ebc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 1 --2- 172.21.15.3:0/143432934 >> v2:172.21.15.92:6800/1750155735 conn(0x7f1ebc06c7c0 0x7f1ebc06ec80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1ec002b8c0 tx=0x7f1ec003d040 comp rx=0 tx=0).stop 2023-10-30T12:32:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 10 monclient: shutdown 2023-10-30T12:32:39.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 1 -- 172.21.15.3:0/143432934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed00b12b0 msgr2=0x7f1ed0143ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 1 --2- 172.21.15.3:0/143432934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed00b12b0 0x7f1ed0143ed0 secure :-1 s=READY pgs=5321 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400b3f0 tx=0x7f1ec400b8b0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ecffff640 1 -- 172.21.15.3:0/143432934 reap_dead start 2023-10-30T12:32:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 1 -- 172.21.15.3:0/143432934 shutdown_connections 2023-10-30T12:32:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 1 -- 172.21.15.3:0/143432934 >> 172.21.15.3:0/143432934 conn(0x7f1ed0065ed0 msgr2=0x7f1ed00b70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:39.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 1 -- 172.21.15.3:0/143432934 shutdown_connections 2023-10-30T12:32:39.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 1 -- 172.21.15.3:0/143432934 wait complete. 2023-10-30T12:32:39.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.006+0000 7f1ed5f47640 1 librados: shutdown 2023-10-30T12:32:39.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-30T12:32:39.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.139+0000 7fbbabe56640 10 monclient: get_monmap_and_config 2023-10-30T12:32:39.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.139+0000 7fbbabe56640 10 monclient: build_initial_monmap 2023-10-30T12:32:39.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.140+0000 7fbbabe56640 10 monclient: monmap: 2023-10-30T12:32:39.142 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:39.142 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:39.143 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:39.140980+0000 2023-10-30T12:32:39.143 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:39.140980+0000 2023-10-30T12:32:39.143 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:39.143 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:39.143 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:39.144 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:39.144 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:39.144 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:39.144 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:39.144 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:39.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.140+0000 7fbbabe56640 1 Processor -- start 2023-10-30T12:32:39.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.140+0000 7fbbabe56640 1 -- start start 2023-10-30T12:32:39.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.140+0000 7fbbabe56640 10 monclient: init 2023-10-30T12:32:39.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.141+0000 7fbbabe56640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:39.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.141+0000 7fbbabe56640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-10-30T12:32:39.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.141+0000 7fbbabe56640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbba40b4350 addr v1:172.21.15.3:6789/0 2023-10-30T12:32:39.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.141+0000 7fbbabe56640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbba413eb50 0x7fbba413efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.141+0000 7fbbabe56640 10 monclient(hunting): picked mon.noname-c con 0x7fbba413eb50 addr v2:172.21.15.92:3300/0 2023-10-30T12:32:39.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.141+0000 7fbbabe56640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbba4144f50 0x7fbba41453a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.141+0000 7fbbabe56640 10 monclient(hunting): picked mon.noname-a con 0x7fbba4144f50 addr v2:172.21.15.3:3300/0 2023-10-30T12:32:39.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.141+0000 7fbbabe56640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba40aa6e0 con 0x7fbba40b4350 2023-10-30T12:32:39.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.141+0000 7fbbabe56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:39.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.141+0000 7fbbabe56640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba40679f0 con 0x7fbba4144f50 2023-10-30T12:32:39.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.141+0000 7fbbabe56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:39.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.141+0000 7fbbabe56640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba40a8bc0 con 0x7fbba413eb50 2023-10-30T12:32:39.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.141+0000 7fbbabe56640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:39.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.141+0000 7fbbabe56640 10 monclient(hunting): authenticate will time out at 2750.355709s 2023-10-30T12:32:39.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.152+0000 7fbba93f1640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbba413eb50 0x7fbba413efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:39.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.152+0000 7fbba93f1640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbba413eb50 0x7fbba413efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44846/0 (socket says 172.21.15.3:44846) 2023-10-30T12:32:39.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.152+0000 7fbba93f1640 1 -- 172.21.15.3:0/1366490009 learned_addr learned my addr 172.21.15.3:0/1366490009 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:39.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.152+0000 7fbba93f1640 10 monclient(hunting): get_auth_request con 0x7fbba413eb50 auth_method 0 2023-10-30T12:32:39.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.152+0000 7fbba93f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:39.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.152+0000 7fbba93f1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:39.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.152+0000 7fbba93f1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:39.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.153+0000 7fbba93f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:39.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.153+0000 7fbba93f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:39.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.153+0000 7fbba93f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:39.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.154+0000 7fbba93f1640 10 monclient(hunting): handle_auth_done global_id 21892 payload 274 2023-10-30T12:32:39.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.154+0000 7fbba93f1640 1 -- 172.21.15.3:0/1366490009 >> v2:172.21.15.3:3300/0 conn(0x7fbba4144f50 msgr2=0x7fbba41453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:39.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.154+0000 7fbba93f1640 1 --2- 172.21.15.3:0/1366490009 >> v2:172.21.15.3:3300/0 conn(0x7fbba4144f50 0x7fbba41453a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.154+0000 7fbba93f1640 1 -- 172.21.15.3:0/1366490009 >> v1:172.21.15.3:6789/0 conn(0x7fbba40b4350 legacy=0x7fbba40a83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:39.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.154+0000 7fbba93f1640 10 monclient: _finish_hunting 0 2023-10-30T12:32:39.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.154+0000 7fbba93f1640 1 monclient: found mon.noname-c 2023-10-30T12:32:39.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.154+0000 7fbba93f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:39.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.154+0000 7fbba93f1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-10-30T12:32:39.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.154+0000 7fbba93f1640 1 -- 172.21.15.3:0/1366490009 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba40adf00 con 0x7fbba413eb50 2023-10-30T12:32:39.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.156+0000 7fbba93f1640 1 --2- 172.21.15.3:0/1366490009 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbba413eb50 0x7fbba413efa0 secure :-1 s=READY pgs=5322 cs=0 l=1 rev1=1 crypto rx=0x7fbba40a9aa0 tx=0x7fbb9402b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60c9c925bc4400dd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:39.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbba8bf0640 1 -- 172.21.15.3:0/1366490009 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb94031750 con 0x7fbba413eb50 2023-10-30T12:32:39.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbba8bf0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:39.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbba8bf0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:32:39.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbba8bf0640 10 monclient: dump: 2023-10-30T12:32:39.162 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:39.163 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:39.163 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.163 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.163 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:39.163 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:39.163 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:39.164 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:39.164 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:39.164 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:39.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbba8bf0640 10 monclient: _finish_auth 0 2023-10-30T12:32:39.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbba8bf0640 10 monclient: _check_auth_tickets 2023-10-30T12:32:39.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbba8bf0640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:39.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbba8bf0640 1 -- 172.21.15.3:0/1366490009 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb940318c0 con 0x7fbba413eb50 2023-10-30T12:32:39.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbba8bf0640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:39.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbba8bf0640 1 -- 172.21.15.3:0/1366490009 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb94031c60 con 0x7fbba413eb50 2023-10-30T12:32:39.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbbabe56640 5 monclient: authenticate success, global_id 21892 2023-10-30T12:32:39.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbbabe56640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:39.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbbabe56640 4 set_mon_vals no callback set 2023-10-30T12:32:39.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbba8bf0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:39.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbba8bf0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:39.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.160+0000 7fbba8bf0640 10 monclient: dump: 2023-10-30T12:32:39.167 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:39.167 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:39.167 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.167 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.167 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:39.167 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:39.168 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:39.168 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:39.168 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:39.168 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:39.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.161+0000 7fbbabe56640 10 monclient: shutdown 2023-10-30T12:32:39.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.161+0000 7fbbabe56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:39.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.161+0000 7fbbabe56640 1 -- 172.21.15.3:0/1366490009 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbba413eb50 msgr2=0x7fbba413efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.161+0000 7fbbabe56640 1 --2- 172.21.15.3:0/1366490009 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbba413eb50 0x7fbba413efa0 secure :-1 s=READY pgs=5322 cs=0 l=1 rev1=1 crypto rx=0x7fbba40a9aa0 tx=0x7fbb9402b7f0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.161+0000 7fbbabe56640 1 -- 172.21.15.3:0/1366490009 shutdown_connections 2023-10-30T12:32:39.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.161+0000 7fbbabe56640 1 --2- 172.21.15.3:0/1366490009 >> v2:172.21.15.3:3300/0 conn(0x7fbba4144f50 0x7fbba41453a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.161+0000 7fbbabe56640 1 --2- 172.21.15.3:0/1366490009 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbba413eb50 0x7fbba413efa0 unknown :-1 s=CLOSED pgs=5322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.161+0000 7fbbabe56640 1 -- 172.21.15.3:0/1366490009 >> 172.21.15.3:0/1366490009 conn(0x7fbba4065ea0 msgr2=0x7fbba40662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:39.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.161+0000 7fbbabe56640 1 -- 172.21.15.3:0/1366490009 shutdown_connections 2023-10-30T12:32:39.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.161+0000 7fbbabe56640 1 -- 172.21.15.3:0/1366490009 wait complete. 2023-10-30T12:32:39.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.162+0000 7fbbabe56640 10 monclient: build_initial_monmap 2023-10-30T12:32:39.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.162+0000 7fbbabe56640 10 monclient: monmap: 2023-10-30T12:32:39.174 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:39.174 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:39.174 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:39.174 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:39.175 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:39.175 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:39.176 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:39.177 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:39.177 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:39.177 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:39.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.162+0000 7fbbabe56640 1 librados: starting msgr at 2023-10-30T12:32:39.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.163+0000 7fbbabe56640 1 librados: starting objecter 2023-10-30T12:32:39.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.163+0000 7fbbabe56640 1 Processor -- start 2023-10-30T12:32:39.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.163+0000 7fbbabe56640 1 -- start start 2023-10-30T12:32:39.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.163+0000 7fbbabe56640 1 librados: setting wanted keys 2023-10-30T12:32:39.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.163+0000 7fbbabe56640 1 librados: calling monclient init 2023-10-30T12:32:39.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.163+0000 7fbbabe56640 10 monclient: init 2023-10-30T12:32:39.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:39.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:32:39.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbba40b4350 0x7fbba406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 10 monclient(hunting): picked mon.noname-c con 0x7fbba40b4350 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:39.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbba413eb50 0x7fbba406d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 10 monclient(hunting): picked mon.noname-b con 0x7fbba413eb50 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:39.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbba4144f50 0x7fbba406ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 10 monclient(hunting): picked mon.noname-a con 0x7fbba4144f50 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:39.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:39.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba4145530 con 0x7fbba4144f50 2023-10-30T12:32:39.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:39.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba40b0850 con 0x7fbba413eb50 2023-10-30T12:32:39.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:39.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba4070450 con 0x7fbba40b4350 2023-10-30T12:32:39.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:39.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbbabe56640 10 monclient(hunting): authenticate will time out at 2750.378243s 2023-10-30T12:32:39.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbba93f1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbba413eb50 0x7fbba406d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:39.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbba93f1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbba413eb50 0x7fbba406d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44854/0 (socket says 172.21.15.3:44854) 2023-10-30T12:32:39.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbba93f1640 1 -- 172.21.15.3:0/510030805 learned_addr learned my addr 172.21.15.3:0/510030805 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:39.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbba93f1640 10 monclient(hunting): get_auth_request con 0x7fbba413eb50 auth_method 0 2023-10-30T12:32:39.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbba93f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:39.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbba93f1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:39.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.164+0000 7fbba93f1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:39.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.165+0000 7fbba9bf2640 1 --2- 172.21.15.3:0/510030805 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbba40b4350 0x7fbba406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:39.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.165+0000 7fbba93f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:39.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.165+0000 7fbba93f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:39.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.165+0000 7fbba93f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:39.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.165+0000 7fbba9bf2640 10 monclient(hunting): get_auth_request con 0x7fbba40b4350 auth_method 0 2023-10-30T12:32:39.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.165+0000 7fbba9bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:39.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.165+0000 7fbba9bf2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:39.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.165+0000 7fbba9bf2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:39.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.166+0000 7fbba9bf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:39.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.166+0000 7fbba9bf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:39.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.166+0000 7fbba9bf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:39.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.167+0000 7fbba93f1640 10 monclient(hunting): handle_auth_done global_id 21895 payload 824 2023-10-30T12:32:39.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.167+0000 7fbba93f1640 1 -- 172.21.15.3:0/510030805 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbba40b4350 msgr2=0x7fbba406dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.167+0000 7fbba93f1640 1 --2- 172.21.15.3:0/510030805 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbba40b4350 0x7fbba406dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.167+0000 7fbba93f1640 1 -- 172.21.15.3:0/510030805 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbba4144f50 msgr2=0x7fbba406ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.167+0000 7fbba93f1640 1 --2- 172.21.15.3:0/510030805 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbba4144f50 0x7fbba406ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.167+0000 7fbba93f1640 10 monclient: _finish_hunting 0 2023-10-30T12:32:39.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.167+0000 7fbba93f1640 1 monclient: found mon.noname-b 2023-10-30T12:32:39.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.167+0000 7fbba93f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:39.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.167+0000 7fbba93f1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:39.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.167+0000 7fbba93f1640 1 -- 172.21.15.3:0/510030805 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba41db240 con 0x7fbba413eb50 2023-10-30T12:32:39.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.173+0000 7fbba93f1640 1 --2- 172.21.15.3:0/510030805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbba413eb50 0x7fbba406d490 secure :-1 s=READY pgs=5323 cs=0 l=1 rev1=1 crypto rx=0x7fbb94000c00 tx=0x7fbb940046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:39.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.173+0000 7fbb9a7fc640 10 client.?.objecter ms_handle_connect 0x7fbba413eb50 2023-10-30T12:32:39.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.173+0000 7fbb9a7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:39.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbb9a7fc640 1 -- 172.21.15.3:0/510030805 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb94033c80 con 0x7fbba413eb50 2023-10-30T12:32:39.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbb9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:39.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbb9a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:39.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbb9a7fc640 10 monclient: dump: 2023-10-30T12:32:39.192 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:39.192 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:39.192 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.192 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.193 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:39.193 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:39.193 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:39.193 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:39.193 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:39.194 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:39.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbb9a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:32:39.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbb9a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:32:39.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbb9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:39.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbb9a7fc640 1 -- 172.21.15.3:0/510030805 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb94033e10 con 0x7fbba413eb50 2023-10-30T12:32:39.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbb9a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:39.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbb9a7fc640 1 -- 172.21.15.3:0/510030805 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9403d6b0 con 0x7fbba413eb50 2023-10-30T12:32:39.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbb9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:39.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbb9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:39.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbb9a7fc640 10 monclient: dump: 2023-10-30T12:32:39.195 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:39.196 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:39.196 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.196 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.196 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:39.196 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:39.197 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:39.197 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:39.197 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:39.197 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:39.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbbabe56640 5 monclient: authenticate success, global_id 21895 2023-10-30T12:32:39.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbbabe56640 10 monclient: _renew_subs 2023-10-30T12:32:39.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbbabe56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:39.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbbabe56640 1 -- 172.21.15.3:0/510030805 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba41db520 con 0x7fbba413eb50 2023-10-30T12:32:39.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.177+0000 7fbb9b7fe640 4 set_mon_vals no callback set 2023-10-30T12:32:39.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.178+0000 7fbbabe56640 10 client.21895.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:39.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.178+0000 7fbbabe56640 10 monclient: _renew_subs 2023-10-30T12:32:39.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.178+0000 7fbbabe56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:39.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.178+0000 7fbbabe56640 1 -- 172.21.15.3:0/510030805 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba41dba40 con 0x7fbba413eb50 2023-10-30T12:32:39.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.178+0000 7fbbabe56640 1 librados: init done 2023-10-30T12:32:39.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.178+0000 7fbbabe56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:39.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.178+0000 7fbbabe56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:39.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.178+0000 7fbbabe56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:39.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.178+0000 7fbbabe56640 1 -- 172.21.15.3:0/510030805 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba404e9b0 con 0x7fbba413eb50 2023-10-30T12:32:39.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.195+0000 7fbb9a7fc640 1 -- 172.21.15.3:0/510030805 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fbb9403a040 con 0x7fbba413eb50 2023-10-30T12:32:39.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.196+0000 7fbb9a7fc640 1 --2- 172.21.15.3:0/510030805 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbb8806c810 0x7fbb8806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.196+0000 7fbba9bf2640 1 --2- 172.21.15.3:0/510030805 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbb8806c810 0x7fbb8806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:39.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.196+0000 7fbba9bf2640 10 monclient: get_auth_request con 0x7fbb8806c810 auth_method 0 2023-10-30T12:32:39.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.197+0000 7fbba9bf2640 1 --2- 172.21.15.3:0/510030805 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbb8806c810 0x7fbb8806ecd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbba0001ef0 tx=0x7fbba0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:39.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.197+0000 7fbb9a7fc640 10 client.21895.objecter ms_handle_connect 0x7fbb8806c810 2023-10-30T12:32:39.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.198+0000 7fbb9a7fc640 1 -- 172.21.15.3:0/510030805 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7fbb9407a870 con 0x7fbba413eb50 2023-10-30T12:32:39.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.198+0000 7fbb9a7fc640 10 client.21895.objecter ms_dispatch 0x7fbba4000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:39.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.198+0000 7fbb9a7fc640 3 client.21895.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:39.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.198+0000 7fbb9a7fc640 3 client.21895.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:39.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.198+0000 7fbb9a7fc640 20 client.21895.objecter dump_active .. 0 homeless 2023-10-30T12:32:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.426+0000 7fbb9a7fc640 1 -- 172.21.15.3:0/510030805 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbb9402f050 con 0x7fbba413eb50 2023-10-30T12:32:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.426+0000 7fbb9a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.426+0000 7fbb9a7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:32:39.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.605+0000 7fbbabe56640 1 -- 172.21.15.3:0/510030805 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fbba41458e0 con 0x7fbb8806c810 2023-10-30T12:32:39.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.605+0000 7fbb9a7fc640 1 -- 172.21.15.3:0/510030805 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbba000fe00 con 0x7fbb8806c810 2023-10-30T12:32:39.607 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 7 to deep-scrub 2023-10-30T12:32:39.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 10 librados: watch_flush enter 2023-10-30T12:32:39.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 10 librados: watch_flush exit 2023-10-30T12:32:39.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 20 client.21895.objecter shutdown clearing up homeless session... 2023-10-30T12:32:39.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 10 client.21895.objecter successfully canceled tick 2023-10-30T12:32:39.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 1 -- 172.21.15.3:0/510030805 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbb8806c810 msgr2=0x7fbb8806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 1 --2- 172.21.15.3:0/510030805 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbb8806c810 0x7fbb8806ecd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbba0001ef0 tx=0x7fbba0009040 comp rx=0 tx=0).stop 2023-10-30T12:32:39.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 10 monclient: shutdown 2023-10-30T12:32:39.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:39.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 1 -- 172.21.15.3:0/510030805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbba413eb50 msgr2=0x7fbba406d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 1 --2- 172.21.15.3:0/510030805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbba413eb50 0x7fbba406d490 secure :-1 s=READY pgs=5323 cs=0 l=1 rev1=1 crypto rx=0x7fbb94000c00 tx=0x7fbb940046c0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 1 -- 172.21.15.3:0/510030805 shutdown_connections 2023-10-30T12:32:39.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 1 --2- 172.21.15.3:0/510030805 >> v2:172.21.15.92:6800/1750155735 conn(0x7fbb8806c810 0x7fbb8806ecd0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 1 --2- 172.21.15.3:0/510030805 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbba4144f50 0x7fbba406ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 1 --2- 172.21.15.3:0/510030805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbba413eb50 0x7fbba406d490 unknown :-1 s=CLOSED pgs=5323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 1 --2- 172.21.15.3:0/510030805 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbba40b4350 0x7fbba406dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 1 -- 172.21.15.3:0/510030805 >> 172.21.15.3:0/510030805 conn(0x7fbba4065ea0 msgr2=0x7fbba413c830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:39.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 1 -- 172.21.15.3:0/510030805 shutdown_connections 2023-10-30T12:32:39.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 1 -- 172.21.15.3:0/510030805 wait complete. 2023-10-30T12:32:39.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.608+0000 7fbbabe56640 1 librados: shutdown 2023-10-30T12:32:39.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:32:39.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.730+0000 7fafd3844640 10 monclient: get_monmap_and_config 2023-10-30T12:32:39.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.730+0000 7fafd3844640 10 monclient: build_initial_monmap 2023-10-30T12:32:39.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.730+0000 7fafd3844640 10 monclient: monmap: 2023-10-30T12:32:39.733 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:39.733 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:39.733 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:32:39.731902+0000 2023-10-30T12:32:39.733 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:32:39.731902+0000 2023-10-30T12:32:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:32:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:32:39.734 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:32:39.735 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:32:39.735 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:32:39.735 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.731+0000 7fafd3844640 1 Processor -- start 2023-10-30T12:32:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.731+0000 7fafd3844640 1 -- start start 2023-10-30T12:32:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.731+0000 7fafd3844640 10 monclient: init 2023-10-30T12:32:39.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.732+0000 7fafd3844640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:39.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.732+0000 7fafd3844640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-10-30T12:32:39.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.732+0000 7fafd3844640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fafcc0ad660 0x7fafcc06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.732+0000 7fafd3844640 10 monclient(hunting): picked mon.noname-b con 0x7fafcc0ad660 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:39.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.732+0000 7fafd3844640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fafcc0ada40 addr v1:172.21.15.92:6789/0 2023-10-30T12:32:39.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.732+0000 7fafd3844640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fafcc141af0 addr v1:172.21.15.3:6789/0 2023-10-30T12:32:39.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.732+0000 7fafd3844640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc070cb0 con 0x7fafcc141af0 2023-10-30T12:32:39.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.732+0000 7fafd3844640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc13c240 con 0x7fafcc0ada40 2023-10-30T12:32:39.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.732+0000 7fafd3844640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:39.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.732+0000 7fafd3844640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc067b90 con 0x7fafcc0ad660 2023-10-30T12:32:39.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.732+0000 7fafd3844640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.732+0000 7fafd3844640 10 monclient(hunting): authenticate will time out at 2750.946530s 2023-10-30T12:32:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.733+0000 7fafd1de1640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fafcc141af0 0x7fafcc0717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:48134/0 (socket says 172.21.15.3:48134) 2023-10-30T12:32:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.733+0000 7fafd1de1640 1 -- 172.21.15.3:0/1838463970 learned_addr learned my addr 172.21.15.3:0/1838463970 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:32:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.733+0000 7fafd15e0640 1 --2- 172.21.15.3:0/1838463970 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fafcc0ad660 0x7fafcc06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:39.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.733+0000 7fafd15e0640 10 monclient(hunting): get_auth_request con 0x7fafcc0ad660 auth_method 0 2023-10-30T12:32:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.733+0000 7fafd15e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.733+0000 7fafd15e0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.733+0000 7fafd15e0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.734+0000 7fafd15e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:39.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.734+0000 7fafd15e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.734+0000 7fafd15e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafcbfff640 1 -- 172.21.15.3:0/1838463970 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827865473 0 0) 0x7fafc0001110 con 0x7fafcc0ada40 2023-10-30T12:32:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafcbfff640 10 monclient(hunting): _negotiate 2023-10-30T12:32:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafcbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafcbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafcbfff640 10 monclient(hunting): my global_id is 21898 2023-10-30T12:32:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafcbfff640 1 -- 172.21.15.3:0/1838463970 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb4003520 con 0x7fafcc0ada40 2023-10-30T12:32:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafcbfff640 1 -- 172.21.15.3:0/1838463970 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880468829 0 0) 0x7fafbc001490 con 0x7fafcc141af0 2023-10-30T12:32:39.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafcbfff640 10 monclient(hunting): _negotiate 2023-10-30T12:32:39.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafcbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:39.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafcbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:39.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafcbfff640 10 monclient(hunting): my global_id is 16695 2023-10-30T12:32:39.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafcbfff640 1 -- 172.21.15.3:0/1838463970 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb4003840 con 0x7fafcc141af0 2023-10-30T12:32:39.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafd15e0640 10 monclient(hunting): handle_auth_done global_id 15614 payload 274 2023-10-30T12:32:39.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafd15e0640 1 -- 172.21.15.3:0/1838463970 >> v1:172.21.15.92:6789/0 conn(0x7fafcc0ada40 legacy=0x7fafcc13bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafd15e0640 1 -- 172.21.15.3:0/1838463970 >> v1:172.21.15.3:6789/0 conn(0x7fafcc141af0 legacy=0x7fafcc0717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafd15e0640 10 monclient: _finish_hunting 0 2023-10-30T12:32:39.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafd15e0640 1 monclient: found mon.noname-b 2023-10-30T12:32:39.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafd15e0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:39.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafd15e0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:32:39.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.735+0000 7fafd15e0640 1 -- 172.21.15.3:0/1838463970 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc071ef0 con 0x7fafcc0ad660 2023-10-30T12:32:39.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.737+0000 7fafd15e0640 1 --2- 172.21.15.3:0/1838463970 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fafcc0ad660 0x7fafcc06ec00 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fafc40097d0 tx=0x7fafc402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b19be4aa1d41f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:39.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafcbfff640 1 -- 172.21.15.3:0/1838463970 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc4031800 con 0x7fafcc0ad660 2023-10-30T12:32:39.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:39.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafcbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafcbfff640 10 monclient: dump: 2023-10-30T12:32:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.753 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:39.753 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:39.753 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:39.753 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:39.753 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:39.753 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:39.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafcbfff640 10 monclient: _finish_auth 0 2023-10-30T12:32:39.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafcbfff640 10 monclient: _check_auth_tickets 2023-10-30T12:32:39.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:39.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafcbfff640 1 -- 172.21.15.3:0/1838463970 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc4031970 con 0x7fafcc0ad660 2023-10-30T12:32:39.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafcbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:39.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafcbfff640 1 -- 172.21.15.3:0/1838463970 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc4031d10 con 0x7fafcc0ad660 2023-10-30T12:32:39.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafd3844640 5 monclient: authenticate success, global_id 15614 2023-10-30T12:32:39.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafd3844640 10 monclient: get_monmap_and_config success 2023-10-30T12:32:39.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafd3844640 4 set_mon_vals no callback set 2023-10-30T12:32:39.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:39.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafcbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:32:39.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.741+0000 7fafcbfff640 10 monclient: dump: 2023-10-30T12:32:39.756 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:39.756 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:39.758 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.742+0000 7fafd3844640 10 monclient: shutdown 2023-10-30T12:32:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.742+0000 7fafd3844640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.742+0000 7fafd3844640 1 -- 172.21.15.3:0/1838463970 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fafcc0ad660 msgr2=0x7fafcc06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.742+0000 7fafd3844640 1 --2- 172.21.15.3:0/1838463970 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fafcc0ad660 0x7fafcc06ec00 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fafc40097d0 tx=0x7fafc402b8a0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.742+0000 7fafd3844640 1 -- 172.21.15.3:0/1838463970 shutdown_connections 2023-10-30T12:32:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.742+0000 7fafd3844640 1 --2- 172.21.15.3:0/1838463970 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fafcc0ad660 0x7fafcc06ec00 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.742+0000 7fafd3844640 1 -- 172.21.15.3:0/1838463970 >> 172.21.15.3:0/1838463970 conn(0x7fafcc0afb80 msgr2=0x7fafcc0aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.742+0000 7fafd3844640 1 -- 172.21.15.3:0/1838463970 shutdown_connections 2023-10-30T12:32:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.742+0000 7fafd3844640 1 -- 172.21.15.3:0/1838463970 wait complete. 2023-10-30T12:32:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.743+0000 7fafd3844640 10 monclient: build_initial_monmap 2023-10-30T12:32:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.743+0000 7fafd3844640 10 monclient: monmap: 2023-10-30T12:32:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:32:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:32:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:32:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:32:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:32:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:32:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:32:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:32:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:32:39.762 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.743+0000 7fafd3844640 1 librados: starting msgr at 2023-10-30T12:32:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.743+0000 7fafd3844640 1 librados: starting objecter 2023-10-30T12:32:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.743+0000 7fafd3844640 1 Processor -- start 2023-10-30T12:32:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.743+0000 7fafd3844640 1 -- start start 2023-10-30T12:32:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.743+0000 7fafd3844640 1 librados: setting wanted keys 2023-10-30T12:32:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.743+0000 7fafd3844640 1 librados: calling monclient init 2023-10-30T12:32:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.743+0000 7fafd3844640 10 monclient: init 2023-10-30T12:32:39.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 10 monclient: _reopen_session rank -1 2023-10-30T12:32:39.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:32:39.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafcc0ad660 0x7fafcc0b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 10 monclient(hunting): picked mon.noname-a con 0x7fafcc0ad660 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:32:39.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fafcc141af0 0x7fafcc13c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 10 monclient(hunting): picked mon.noname-c con 0x7fafcc141af0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:32:39.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafcc13f6c0 0x7fafcc13fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 10 monclient(hunting): picked mon.noname-b con 0x7fafcc13f6c0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:32:39.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:39.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc06e620 con 0x7fafcc0ad660 2023-10-30T12:32:39.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:39.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc1418c0 con 0x7fafcc13f6c0 2023-10-30T12:32:39.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 10 monclient(hunting): start opening mon connection 2023-10-30T12:32:39.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc13d1d0 con 0x7fafcc141af0 2023-10-30T12:32:39.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 10 monclient(hunting): _renew_subs 2023-10-30T12:32:39.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd3844640 10 monclient(hunting): authenticate will time out at 2750.958456s 2023-10-30T12:32:39.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd15e0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafcc0ad660 0x7fafcc0b24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:39.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.744+0000 7fafd1de1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafcc13f6c0 0x7fafcc13fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:39.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd0ddf640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fafcc141af0 0x7fafcc13c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:39.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd1de1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafcc13f6c0 0x7fafcc13fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44862/0 (socket says 172.21.15.3:44862) 2023-10-30T12:32:39.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd1de1640 1 -- 172.21.15.3:0/132564326 learned_addr learned my addr 172.21.15.3:0/132564326 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:32:39.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd1de1640 10 monclient(hunting): get_auth_request con 0x7fafcc13f6c0 auth_method 0 2023-10-30T12:32:39.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd1de1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:39.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd1de1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:39.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd1de1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd0ddf640 10 monclient(hunting): get_auth_request con 0x7fafcc141af0 auth_method 0 2023-10-30T12:32:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd0ddf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd0ddf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd0ddf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd15e0640 10 monclient(hunting): get_auth_request con 0x7fafcc0ad660 auth_method 0 2023-10-30T12:32:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd15e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:32:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd15e0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:32:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd15e0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:32:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd1de1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd1de1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd1de1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd0ddf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd0ddf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd0ddf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd15e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:32:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.745+0000 7fafd15e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:32:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.746+0000 7fafd15e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:32:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.748+0000 7fafd1de1640 10 monclient(hunting): handle_auth_done global_id 21901 payload 824 2023-10-30T12:32:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.748+0000 7fafd1de1640 1 -- 172.21.15.3:0/132564326 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fafcc141af0 msgr2=0x7fafcc13c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.748+0000 7fafd1de1640 1 --2- 172.21.15.3:0/132564326 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fafcc141af0 0x7fafcc13c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.748+0000 7fafd1de1640 1 -- 172.21.15.3:0/132564326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafcc0ad660 msgr2=0x7fafcc0b24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.748+0000 7fafd1de1640 1 --2- 172.21.15.3:0/132564326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafcc0ad660 0x7fafcc0b24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.748+0000 7fafd1de1640 10 monclient: _finish_hunting 0 2023-10-30T12:32:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.748+0000 7fafd1de1640 1 monclient: found mon.noname-b 2023-10-30T12:32:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.748+0000 7fafd1de1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:32:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.748+0000 7fafd1de1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:32:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.748+0000 7fafd1de1640 1 -- 172.21.15.3:0/132564326 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc13d9a0 con 0x7fafcc13f6c0 2023-10-30T12:32:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.750+0000 7fafd1de1640 1 --2- 172.21.15.3:0/132564326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafcc13f6c0 0x7fafcc13fb20 secure :-1 s=READY pgs=5325 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00cb40 tx=0x7fafbc00a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.750+0000 7fafc9ffb640 10 client.?.objecter ms_handle_connect 0x7fafcc13f6c0 2023-10-30T12:32:39.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.750+0000 7fafc9ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:32:39.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafc9ffb640 1 -- 172.21.15.3:0/132564326 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc00aca0 con 0x7fafcc13f6c0 2023-10-30T12:32:39.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:39.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafc9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:32:39.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafc9ffb640 10 monclient: dump: 2023-10-30T12:32:39.776 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:39.777 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:39.777 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.777 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.777 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:39.778 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:39.778 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:39.778 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:39.778 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:39.779 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:39.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafc9ffb640 10 monclient: _finish_auth 0 2023-10-30T12:32:39.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafc9ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:32:39.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:32:39.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafc9ffb640 1 -- 172.21.15.3:0/132564326 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafbc00e040 con 0x7fafcc13f6c0 2023-10-30T12:32:39.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafd3844640 5 monclient: authenticate success, global_id 21901 2023-10-30T12:32:39.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafc9ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:32:39.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafd3844640 10 monclient: _renew_subs 2023-10-30T12:32:39.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafc9ffb640 1 -- 172.21.15.3:0/132564326 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc012600 con 0x7fafcc13f6c0 2023-10-30T12:32:39.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafd3844640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:39.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafd3844640 1 -- 172.21.15.3:0/132564326 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafcc13ce50 con 0x7fafcc13f6c0 2023-10-30T12:32:39.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:32:39.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafc9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:32:39.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafcb7fe640 4 set_mon_vals no callback set 2023-10-30T12:32:39.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafc9ffb640 10 monclient: dump: 2023-10-30T12:32:39.782 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:32:39.782 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:32:39.782 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.782 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:32:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:32:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:32:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:32:39.783 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:32:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:32:39.784 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:32:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafd3844640 10 client.21901.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:32:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafd3844640 10 monclient: _renew_subs 2023-10-30T12:32:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafd3844640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafd3844640 1 -- 172.21.15.3:0/132564326 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafcc140300 con 0x7fafcc13f6c0 2023-10-30T12:32:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.754+0000 7fafd3844640 1 librados: init done 2023-10-30T12:32:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.755+0000 7fafd3844640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.755+0000 7fafd3844640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:39.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.755+0000 7fafd3844640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:32:39.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.755+0000 7fafd3844640 1 -- 172.21.15.3:0/132564326 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafcc06e020 con 0x7fafcc13f6c0 2023-10-30T12:32:39.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.996+0000 7fafc9ffb640 1 -- 172.21.15.3:0/132564326 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fafbc007070 con 0x7fafcc13f6c0 2023-10-30T12:32:39.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.996+0000 7fafc9ffb640 1 --2- 172.21.15.3:0/132564326 >> v2:172.21.15.92:6800/1750155735 conn(0x7fafb806c810 0x7fafb806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.996+0000 7fafd15e0640 1 --2- 172.21.15.3:0/132564326 >> v2:172.21.15.92:6800/1750155735 conn(0x7fafb806c810 0x7fafb806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:39.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.996+0000 7fafd15e0640 10 monclient: get_auth_request con 0x7fafb806c810 auth_method 0 2023-10-30T12:32:39.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.997+0000 7fafd15e0640 1 --2- 172.21.15.3:0/132564326 >> v2:172.21.15.92:6800/1750155735 conn(0x7fafb806c810 0x7fafb806ecd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fafc4036000 tx=0x7fafc403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:39.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.997+0000 7fafc9ffb640 10 client.21901.objecter ms_handle_connect 0x7fafb806c810 2023-10-30T12:32:40.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.998+0000 7fafc9ffb640 1 -- 172.21.15.3:0/132564326 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7fafbc058220 con 0x7fafcc13f6c0 2023-10-30T12:32:40.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.998+0000 7fafc9ffb640 10 client.21901.objecter ms_dispatch 0x7fafcc000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:32:40.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.998+0000 7fafc9ffb640 3 client.21901.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:32:40.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.998+0000 7fafc9ffb640 3 client.21901.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:32:40.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:39.998+0000 7fafc9ffb640 20 client.21901.objecter dump_active .. 0 homeless 2023-10-30T12:32:40.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.020+0000 7fafc9ffb640 1 -- 172.21.15.3:0/132564326 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fafbc05c2c0 con 0x7fafcc13f6c0 2023-10-30T12:32:40.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.020+0000 7fafc9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:32:40.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.020+0000 7fafc9ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:32:40.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.213+0000 7fafd3844640 1 -- 172.21.15.3:0/132564326 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fafcc071300 con 0x7fafb806c810 2023-10-30T12:32:40.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.218+0000 7fafc9ffb640 1 -- 172.21.15.3:0/132564326 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+150843 (secure 0 0 0) 0x7fafc4031750 con 0x7fafb806c810 2023-10-30T12:32:40.221 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:32:40.222 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-10-30T12:32:40.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.220+0000 7fafd3844640 10 librados: watch_flush enter 2023-10-30T12:32:40.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.220+0000 7fafd3844640 10 librados: watch_flush exit 2023-10-30T12:32:40.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.220+0000 7fafd3844640 20 client.21901.objecter shutdown clearing up homeless session... 2023-10-30T12:32:40.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.220+0000 7fafd3844640 10 client.21901.objecter successfully canceled tick 2023-10-30T12:32:40.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.221+0000 7fafd3844640 1 -- 172.21.15.3:0/132564326 >> v2:172.21.15.92:6800/1750155735 conn(0x7fafb806c810 msgr2=0x7fafb806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:40.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.221+0000 7fafd3844640 1 --2- 172.21.15.3:0/132564326 >> v2:172.21.15.92:6800/1750155735 conn(0x7fafb806c810 0x7fafb806ecd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fafc4036000 tx=0x7fafc403c040 comp rx=0 tx=0).stop 2023-10-30T12:32:40.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.221+0000 7fafd3844640 10 monclient: shutdown 2023-10-30T12:32:40.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.221+0000 7fafd3844640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:32:40.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.221+0000 7fafd3844640 1 -- 172.21.15.3:0/132564326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafcc13f6c0 msgr2=0x7fafcc13fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:40.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.221+0000 7fafd3844640 1 --2- 172.21.15.3:0/132564326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafcc13f6c0 0x7fafcc13fb20 secure :-1 s=READY pgs=5325 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00cb40 tx=0x7fafbc00a060 comp rx=0 tx=0).stop 2023-10-30T12:32:40.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.221+0000 7fafd3844640 1 -- 172.21.15.3:0/132564326 shutdown_connections 2023-10-30T12:32:40.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.221+0000 7fafd3844640 1 --2- 172.21.15.3:0/132564326 >> v2:172.21.15.92:6800/1750155735 conn(0x7fafb806c810 0x7fafb806ecd0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:40.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.221+0000 7fafd3844640 1 --2- 172.21.15.3:0/132564326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafcc13f6c0 0x7fafcc13fb20 unknown :-1 s=CLOSED pgs=5325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:40.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.221+0000 7fafd3844640 1 --2- 172.21.15.3:0/132564326 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fafcc141af0 0x7fafcc13c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:40.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.221+0000 7fafd3844640 1 --2- 172.21.15.3:0/132564326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafcc0ad660 0x7fafcc0b24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:40.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.221+0000 7fafd3844640 1 -- 172.21.15.3:0/132564326 >> 172.21.15.3:0/132564326 conn(0x7fafcc0afb80 msgr2=0x7fafcc0ae780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:40.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.221+0000 7fafd3844640 1 -- 172.21.15.3:0/132564326 shutdown_connections 2023-10-30T12:32:40.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.221+0000 7fafd3844640 1 -- 172.21.15.3:0/132564326 wait complete. 2023-10-30T12:32:40.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:32:40.221+0000 7fafd3844640 1 librados: shutdown 2023-10-30T12:32:40.233 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1583,"stamp":"2023-10-30T12:32:40.107992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":579720,"kb_used_data":52976,"kb_used_omap":12,"kb_used_meta":525491,"kb_avail":754395000,"statfs":{"total":773094113280,"available":772500480000,"internally_reserved":0,"allocated":54247424,"data_stored":26382970,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12743,"internal_metadata":538103353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":86,"apply_latency_ms":86,"commit_latency_ns":86000000,"apply_latency_ns":86000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001897"},"pg_stats":[{"pgid":"24.4","version":"0'0","reported_seq":853,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:32:28.651496+0000","last_change":"2023-10-30T12:11:50.816793+0000","last_active":"2023-10-30T12:32:28.651496+0000","last_peered":"2023-10-30T12:32:28.651496+0000","last_clean":"2023-10-30T12:32:28.651496+0000","last_became_active":"2023-10-30T12:11:50.811294+0000","last_became_peered":"2023-10-30T12:11:50.811294+0000","last_unstale":"2023-10-30T12:32:28.651496+0000","last_undegraded":"2023-10-30T12:32:28.651496+0000","last_fullsized":"2023-10-30T12:32:28.651496+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:34:55.305107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":885,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.033802+0000","last_change":"2023-10-30T12:11:58.574669+0000","last_active":"2023-10-30T12:32:33.033802+0000","last_peered":"2023-10-30T12:32:33.033802+0000","last_clean":"2023-10-30T12:32:33.033802+0000","last_became_active":"2023-10-30T12:11:58.558569+0000","last_became_peered":"2023-10-30T12:11:58.558569+0000","last_unstale":"2023-10-30T12:32:33.033802+0000","last_undegraded":"2023-10-30T12:32:33.033802+0000","last_fullsized":"2023-10-30T12:32:33.033802+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":885,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.038677+0000","last_change":"2023-10-30T12:11:58.500558+0000","last_active":"2023-10-30T12:32:33.038677+0000","last_peered":"2023-10-30T12:32:33.038677+0000","last_clean":"2023-10-30T12:32:33.038677+0000","last_became_active":"2023-10-30T12:11:58.495252+0000","last_became_peered":"2023-10-30T12:11:58.495252+0000","last_unstale":"2023-10-30T12:32:33.038677+0000","last_undegraded":"2023-10-30T12:32:33.038677+0000","last_fullsized":"2023-10-30T12:32:33.038677+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":869,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.049256+0000","last_change":"2023-10-30T12:12:25.951506+0000","last_active":"2023-10-30T12:32:33.049256+0000","last_peered":"2023-10-30T12:32:33.049256+0000","last_clean":"2023-10-30T12:32:33.049256+0000","last_became_active":"2023-10-30T12:12:25.941872+0000","last_became_peered":"2023-10-30T12:12:25.941872+0000","last_unstale":"2023-10-30T12:32:33.049256+0000","last_undegraded":"2023-10-30T12:32:33.049256+0000","last_fullsized":"2023-10-30T12:32:33.049256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":871,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.980864+0000","last_change":"2023-10-30T12:11:51.588272+0000","last_active":"2023-10-30T12:32:33.980864+0000","last_peered":"2023-10-30T12:32:33.980864+0000","last_clean":"2023-10-30T12:32:33.980864+0000","last_became_active":"2023-10-30T12:11:51.581699+0000","last_became_peered":"2023-10-30T12:11:51.581699+0000","last_unstale":"2023-10-30T12:32:33.980864+0000","last_undegraded":"2023-10-30T12:32:33.980864+0000","last_fullsized":"2023-10-30T12:32:33.980864+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1","version":"50'7","reported_seq":869,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:34.444518+0000","last_change":"2023-10-30T12:12:02.329571+0000","last_active":"2023-10-30T12:32:34.444518+0000","last_peered":"2023-10-30T12:32:34.444518+0000","last_clean":"2023-10-30T12:32:34.444518+0000","last_became_active":"2023-10-30T12:12:02.320185+0000","last_became_peered":"2023-10-30T12:12:02.320185+0000","last_unstale":"2023-10-30T12:32:34.444518+0000","last_undegraded":"2023-10-30T12:32:34.444518+0000","last_fullsized":"2023-10-30T12:32:34.444518+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":856,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:34.448282+0000","last_change":"2023-10-30T12:12:02.278839+0000","last_active":"2023-10-30T12:32:34.448282+0000","last_peered":"2023-10-30T12:32:34.448282+0000","last_clean":"2023-10-30T12:32:34.448282+0000","last_became_active":"2023-10-30T12:12:02.269307+0000","last_became_peered":"2023-10-30T12:12:02.269307+0000","last_unstale":"2023-10-30T12:32:34.448282+0000","last_undegraded":"2023-10-30T12:32:34.448282+0000","last_fullsized":"2023-10-30T12:32:34.448282+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":855,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:32.991782+0000","last_change":"2023-10-30T12:11:55.586707+0000","last_active":"2023-10-30T12:32:32.991782+0000","last_peered":"2023-10-30T12:32:32.991782+0000","last_clean":"2023-10-30T12:32:32.991782+0000","last_became_active":"2023-10-30T12:11:55.581296+0000","last_became_peered":"2023-10-30T12:11:55.581296+0000","last_unstale":"2023-10-30T12:32:32.991782+0000","last_undegraded":"2023-10-30T12:32:32.991782+0000","last_fullsized":"2023-10-30T12:32:32.991782+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":869,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.054687+0000","last_change":"2023-10-30T12:12:25.845763+0000","last_active":"2023-10-30T12:32:33.054687+0000","last_peered":"2023-10-30T12:32:33.054687+0000","last_clean":"2023-10-30T12:32:33.054687+0000","last_became_active":"2023-10-30T12:12:25.833774+0000","last_became_peered":"2023-10-30T12:12:25.833774+0000","last_unstale":"2023-10-30T12:32:33.054687+0000","last_undegraded":"2023-10-30T12:32:33.054687+0000","last_fullsized":"2023-10-30T12:32:33.054687+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":869,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.039174+0000","last_change":"2023-10-30T12:12:09.181066+0000","last_active":"2023-10-30T12:32:33.039174+0000","last_peered":"2023-10-30T12:32:33.039174+0000","last_clean":"2023-10-30T12:32:33.039174+0000","last_became_active":"2023-10-30T12:12:09.094001+0000","last_became_peered":"2023-10-30T12:12:09.094001+0000","last_unstale":"2023-10-30T12:32:33.039174+0000","last_undegraded":"2023-10-30T12:32:33.039174+0000","last_fullsized":"2023-10-30T12:32:33.039174+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":870,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.996727+0000","last_change":"2023-10-30T12:12:04.088614+0000","last_active":"2023-10-30T12:32:33.996727+0000","last_peered":"2023-10-30T12:32:33.996727+0000","last_clean":"2023-10-30T12:32:33.996727+0000","last_became_active":"2023-10-30T12:12:04.077824+0000","last_became_peered":"2023-10-30T12:12:04.077824+0000","last_unstale":"2023-10-30T12:32:33.996727+0000","last_undegraded":"2023-10-30T12:32:33.996727+0000","last_fullsized":"2023-10-30T12:32:33.996727+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":851,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:32:28.656679+0000","last_change":"2023-10-30T12:11:50.993004+0000","last_active":"2023-10-30T12:32:28.656679+0000","last_peered":"2023-10-30T12:32:28.656679+0000","last_clean":"2023-10-30T12:32:28.656679+0000","last_became_active":"2023-10-30T12:11:50.987299+0000","last_became_peered":"2023-10-30T12:11:50.987299+0000","last_unstale":"2023-10-30T12:32:28.656679+0000","last_undegraded":"2023-10-30T12:32:28.656679+0000","last_fullsized":"2023-10-30T12:32:28.656679+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:02:34.008030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":852,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:32:28.661450+0000","last_change":"2023-10-30T12:11:51.028381+0000","last_active":"2023-10-30T12:32:28.661450+0000","last_peered":"2023-10-30T12:32:28.661450+0000","last_clean":"2023-10-30T12:32:28.661450+0000","last_became_active":"2023-10-30T12:11:51.023075+0000","last_became_peered":"2023-10-30T12:11:51.023075+0000","last_unstale":"2023-10-30T12:32:28.661450+0000","last_undegraded":"2023-10-30T12:32:28.661450+0000","last_fullsized":"2023-10-30T12:32:28.661450+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:23:10.299251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":876,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.047578+0000","last_change":"2023-10-30T12:10:17.745135+0000","last_active":"2023-10-30T12:32:33.047578+0000","last_peered":"2023-10-30T12:32:33.047578+0000","last_clean":"2023-10-30T12:32:33.047578+0000","last_became_active":"2023-10-30T12:10:17.738241+0000","last_became_peered":"2023-10-30T12:10:17.738241+0000","last_unstale":"2023-10-30T12:32:33.047578+0000","last_undegraded":"2023-10-30T12:32:33.047578+0000","last_fullsized":"2023-10-30T12:32:33.047578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":862,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.819336+0000","last_change":"2023-10-30T12:11:52.264262+0000","last_active":"2023-10-30T12:32:33.819336+0000","last_peered":"2023-10-30T12:32:33.819336+0000","last_clean":"2023-10-30T12:32:33.819336+0000","last_became_active":"2023-10-30T12:11:52.257573+0000","last_became_peered":"2023-10-30T12:11:52.257573+0000","last_unstale":"2023-10-30T12:32:33.819336+0000","last_undegraded":"2023-10-30T12:32:33.819336+0000","last_fullsized":"2023-10-30T12:32:33.819336+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.999374+0000","last_change":"2023-10-30T12:10:17.790588+0000","last_active":"2023-10-30T12:32:33.999374+0000","last_peered":"2023-10-30T12:32:33.999374+0000","last_clean":"2023-10-30T12:32:33.999374+0000","last_became_active":"2023-10-30T12:10:17.763227+0000","last_became_peered":"2023-10-30T12:10:17.763227+0000","last_unstale":"2023-10-30T12:32:33.999374+0000","last_undegraded":"2023-10-30T12:32:33.999374+0000","last_fullsized":"2023-10-30T12:32:33.999374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":886,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:37.760047+0000","last_change":"2023-10-30T12:11:55.621845+0000","last_active":"2023-10-30T12:32:37.760047+0000","last_peered":"2023-10-30T12:32:37.760047+0000","last_clean":"2023-10-30T12:32:37.760047+0000","last_became_active":"2023-10-30T12:11:55.613173+0000","last_became_peered":"2023-10-30T12:11:55.613173+0000","last_unstale":"2023-10-30T12:32:37.760047+0000","last_undegraded":"2023-10-30T12:32:37.760047+0000","last_fullsized":"2023-10-30T12:32:37.760047+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":876,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.036412+0000","last_change":"2023-10-30T12:10:16.912669+0000","last_active":"2023-10-30T12:32:33.036412+0000","last_peered":"2023-10-30T12:32:33.036412+0000","last_clean":"2023-10-30T12:32:33.036412+0000","last_became_active":"2023-10-30T12:10:16.907205+0000","last_became_peered":"2023-10-30T12:10:16.907205+0000","last_unstale":"2023-10-30T12:32:33.036412+0000","last_undegraded":"2023-10-30T12:32:33.036412+0000","last_fullsized":"2023-10-30T12:32:33.036412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":871,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:37.904809+0000","last_change":"2023-10-30T12:32:37.905916+0000","last_active":"2023-10-30T12:32:37.904809+0000","last_peered":"2023-10-30T12:32:37.904809+0000","last_clean":"2023-10-30T12:32:37.904809+0000","last_became_active":"2023-10-30T12:11:52.909795+0000","last_became_peered":"2023-10-30T12:11:52.909795+0000","last_unstale":"2023-10-30T12:32:37.904809+0000","last_undegraded":"2023-10-30T12:32:37.904809+0000","last_fullsized":"2023-10-30T12:32:37.904809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:37.903361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:37.903361+0000","last_clean_scrub_stamp":"2023-10-30T12:32:37.903361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:22.724580+0000","scrub_duration":0.049141183999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":900,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:38.541118+0000","last_change":"2023-10-30T12:32:38.545483+0000","last_active":"2023-10-30T12:32:38.541118+0000","last_peered":"2023-10-30T12:32:38.541118+0000","last_clean":"2023-10-30T12:32:38.541118+0000","last_became_active":"2023-10-30T12:10:17.582475+0000","last_became_peered":"2023-10-30T12:10:17.582475+0000","last_unstale":"2023-10-30T12:32:38.541118+0000","last_undegraded":"2023-10-30T12:32:38.541118+0000","last_fullsized":"2023-10-30T12:32:38.541118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:38.539673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:38.539673+0000","last_clean_scrub_stamp":"2023-10-30T12:32:38.539673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:59:45.700659+0000","scrub_duration":0.055551556000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":861,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.223835+0000","last_change":"2023-10-30T12:11:51.597894+0000","last_active":"2023-10-30T12:32:33.223835+0000","last_peered":"2023-10-30T12:32:33.223835+0000","last_clean":"2023-10-30T12:32:33.223835+0000","last_became_active":"2023-10-30T12:11:51.581905+0000","last_became_peered":"2023-10-30T12:11:51.581905+0000","last_unstale":"2023-10-30T12:32:33.223835+0000","last_undegraded":"2023-10-30T12:32:33.223835+0000","last_fullsized":"2023-10-30T12:32:33.223835+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":876,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.027950+0000","last_change":"2023-10-30T12:10:17.562627+0000","last_active":"2023-10-30T12:32:33.027950+0000","last_peered":"2023-10-30T12:32:33.027950+0000","last_clean":"2023-10-30T12:32:33.027950+0000","last_became_active":"2023-10-30T12:10:17.553523+0000","last_became_peered":"2023-10-30T12:10:17.553523+0000","last_unstale":"2023-10-30T12:32:33.027950+0000","last_undegraded":"2023-10-30T12:32:33.027950+0000","last_fullsized":"2023-10-30T12:32:33.027950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":852,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:32:28.644666+0000","last_change":"2023-10-30T12:11:50.971305+0000","last_active":"2023-10-30T12:32:28.644666+0000","last_peered":"2023-10-30T12:32:28.644666+0000","last_clean":"2023-10-30T12:32:28.644666+0000","last_became_active":"2023-10-30T12:11:50.965758+0000","last_became_peered":"2023-10-30T12:11:50.965758+0000","last_unstale":"2023-10-30T12:32:28.644666+0000","last_undegraded":"2023-10-30T12:32:28.644666+0000","last_fullsized":"2023-10-30T12:32:28.644666+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:57:57.631020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":860,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:32:28.634095+0000","last_change":"2023-10-30T12:10:16.841521+0000","last_active":"2023-10-30T12:32:28.634095+0000","last_peered":"2023-10-30T12:32:28.634095+0000","last_clean":"2023-10-30T12:32:28.634095+0000","last_became_active":"2023-10-30T12:10:16.811711+0000","last_became_peered":"2023-10-30T12:10:16.811711+0000","last_unstale":"2023-10-30T12:32:28.634095+0000","last_undegraded":"2023-10-30T12:32:28.634095+0000","last_fullsized":"2023-10-30T12:32:28.634095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:15.050608+0000","last_clean_scrub_stamp":"2023-10-30T12:10:15.050608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:57.059004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":852,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:32:28.639920+0000","last_change":"2023-10-30T12:12:02.414741+0000","last_active":"2023-10-30T12:32:28.639920+0000","last_peered":"2023-10-30T12:32:28.639920+0000","last_clean":"2023-10-30T12:32:28.639920+0000","last_became_active":"2023-10-30T12:12:02.408646+0000","last_became_peered":"2023-10-30T12:12:02.408646+0000","last_unstale":"2023-10-30T12:32:28.639920+0000","last_undegraded":"2023-10-30T12:32:28.639920+0000","last_fullsized":"2023-10-30T12:32:28.639920+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:48:04.995634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":885,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:36.670764+0000","last_change":"2023-10-30T12:32:36.671858+0000","last_active":"2023-10-30T12:32:36.670764+0000","last_peered":"2023-10-30T12:32:36.670764+0000","last_clean":"2023-10-30T12:32:36.670764+0000","last_became_active":"2023-10-30T12:10:16.849940+0000","last_became_peered":"2023-10-30T12:10:16.849940+0000","last_unstale":"2023-10-30T12:32:36.670764+0000","last_undegraded":"2023-10-30T12:32:36.670764+0000","last_fullsized":"2023-10-30T12:32:36.670764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:36.669332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:36.669332+0000","last_clean_scrub_stamp":"2023-10-30T12:32:36.669332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:53:28.553745+0000","scrub_duration":0.090791832000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":870,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:36.882845+0000","last_change":"2023-10-30T12:32:36.883963+0000","last_active":"2023-10-30T12:32:36.882845+0000","last_peered":"2023-10-30T12:32:36.882845+0000","last_clean":"2023-10-30T12:32:36.882845+0000","last_became_active":"2023-10-30T12:11:53.066917+0000","last_became_peered":"2023-10-30T12:11:53.066917+0000","last_unstale":"2023-10-30T12:32:36.882845+0000","last_undegraded":"2023-10-30T12:32:36.882845+0000","last_fullsized":"2023-10-30T12:32:36.882845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:36.881387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:36.881387+0000","last_clean_scrub_stamp":"2023-10-30T12:32:36.881387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:33:50.617590+0000","scrub_duration":0.073083355000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":884,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:38.638352+0000","last_change":"2023-10-30T12:32:38.640066+0000","last_active":"2023-10-30T12:32:38.638352+0000","last_peered":"2023-10-30T12:32:38.638352+0000","last_clean":"2023-10-30T12:32:38.638352+0000","last_became_active":"2023-10-30T12:10:17.153120+0000","last_became_peered":"2023-10-30T12:10:17.153120+0000","last_unstale":"2023-10-30T12:32:38.638352+0000","last_undegraded":"2023-10-30T12:32:38.638352+0000","last_fullsized":"2023-10-30T12:32:38.638352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:38.636075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:38.636075+0000","last_clean_scrub_stamp":"2023-10-30T12:32:38.636075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:06:44.125261+0000","scrub_duration":0.047870970999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":969,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:32.977046+0000","last_change":"2023-10-30T12:10:09.781745+0000","last_active":"2023-10-30T12:32:32.977046+0000","last_peered":"2023-10-30T12:32:32.977046+0000","last_clean":"2023-10-30T12:32:32.977046+0000","last_became_active":"2023-10-30T12:10:09.748373+0000","last_became_peered":"2023-10-30T12:10:09.748373+0000","last_unstale":"2023-10-30T12:32:32.977046+0000","last_undegraded":"2023-10-30T12:32:32.977046+0000","last_fullsized":"2023-10-30T12:32:32.977046+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:10:07.000545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:10:07.000545+0000","last_clean_scrub_stamp":"2023-10-30T12:10:07.000545+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":862,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.802305+0000","last_change":"2023-10-30T12:12:03.636874+0000","last_active":"2023-10-30T12:32:33.802305+0000","last_peered":"2023-10-30T12:32:33.802305+0000","last_clean":"2023-10-30T12:32:33.802305+0000","last_became_active":"2023-10-30T12:12:03.631410+0000","last_became_peered":"2023-10-30T12:12:03.631410+0000","last_unstale":"2023-10-30T12:32:33.802305+0000","last_undegraded":"2023-10-30T12:32:33.802305+0000","last_fullsized":"2023-10-30T12:32:33.802305+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":870,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:32:28.647848+0000","last_change":"2023-10-30T12:12:01.935080+0000","last_active":"2023-10-30T12:32:28.647848+0000","last_peered":"2023-10-30T12:32:28.647848+0000","last_clean":"2023-10-30T12:32:28.647848+0000","last_became_active":"2023-10-30T12:12:01.926913+0000","last_became_peered":"2023-10-30T12:12:01.926913+0000","last_unstale":"2023-10-30T12:32:28.647848+0000","last_undegraded":"2023-10-30T12:32:28.647848+0000","last_fullsized":"2023-10-30T12:32:28.647848+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:08:28.436909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":852,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:32:28.631728+0000","last_change":"2023-10-30T12:12:02.114160+0000","last_active":"2023-10-30T12:32:28.631728+0000","last_peered":"2023-10-30T12:32:28.631728+0000","last_clean":"2023-10-30T12:32:28.631728+0000","last_became_active":"2023-10-30T12:12:02.108733+0000","last_became_peered":"2023-10-30T12:12:02.108733+0000","last_unstale":"2023-10-30T12:32:28.631728+0000","last_undegraded":"2023-10-30T12:32:28.631728+0000","last_fullsized":"2023-10-30T12:32:28.631728+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:57.228464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":861,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.222166+0000","last_change":"2023-10-30T12:11:52.314662+0000","last_active":"2023-10-30T12:32:33.222166+0000","last_peered":"2023-10-30T12:32:33.222166+0000","last_clean":"2023-10-30T12:32:33.222166+0000","last_became_active":"2023-10-30T12:11:52.304910+0000","last_became_peered":"2023-10-30T12:11:52.304910+0000","last_unstale":"2023-10-30T12:32:33.222166+0000","last_undegraded":"2023-10-30T12:32:33.222166+0000","last_fullsized":"2023-10-30T12:32:33.222166+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":868,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.043680+0000","last_change":"2023-10-30T12:12:10.797605+0000","last_active":"2023-10-30T12:32:33.043680+0000","last_peered":"2023-10-30T12:32:33.043680+0000","last_clean":"2023-10-30T12:32:33.043680+0000","last_became_active":"2023-10-30T12:12:10.791950+0000","last_became_peered":"2023-10-30T12:12:10.791950+0000","last_unstale":"2023-10-30T12:32:33.043680+0000","last_undegraded":"2023-10-30T12:32:33.043680+0000","last_fullsized":"2023-10-30T12:32:33.043680+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":870,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:32:28.656685+0000","last_change":"2023-10-30T12:11:56.058505+0000","last_active":"2023-10-30T12:32:28.656685+0000","last_peered":"2023-10-30T12:32:28.656685+0000","last_clean":"2023-10-30T12:32:28.656685+0000","last_became_active":"2023-10-30T12:11:56.049790+0000","last_became_peered":"2023-10-30T12:11:56.049790+0000","last_unstale":"2023-10-30T12:32:28.656685+0000","last_undegraded":"2023-10-30T12:32:28.656685+0000","last_fullsized":"2023-10-30T12:32:28.656685+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:10.872206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":852,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:32:28.659774+0000","last_change":"2023-10-30T12:12:02.443381+0000","last_active":"2023-10-30T12:32:28.659774+0000","last_peered":"2023-10-30T12:32:28.659774+0000","last_clean":"2023-10-30T12:32:28.659774+0000","last_became_active":"2023-10-30T12:12:02.435431+0000","last_became_peered":"2023-10-30T12:12:02.435431+0000","last_unstale":"2023-10-30T12:32:28.659774+0000","last_undegraded":"2023-10-30T12:32:28.659774+0000","last_fullsized":"2023-10-30T12:32:28.659774+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:04.786017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":852,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:32:28.653153+0000","last_change":"2023-10-30T12:11:51.298662+0000","last_active":"2023-10-30T12:32:28.653153+0000","last_peered":"2023-10-30T12:32:28.653153+0000","last_clean":"2023-10-30T12:32:28.653153+0000","last_became_active":"2023-10-30T12:11:51.289446+0000","last_became_peered":"2023-10-30T12:11:51.289446+0000","last_unstale":"2023-10-30T12:32:28.653153+0000","last_undegraded":"2023-10-30T12:32:28.653153+0000","last_fullsized":"2023-10-30T12:32:28.653153+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:03:40.836674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":853,"reported_epoch":899,"state":"active+clean","last_fresh":"2023-10-30T12:32:28.646963+0000","last_change":"2023-10-30T12:11:53.122439+0000","last_active":"2023-10-30T12:32:28.646963+0000","last_peered":"2023-10-30T12:32:28.646963+0000","last_clean":"2023-10-30T12:32:28.646963+0000","last_became_active":"2023-10-30T12:11:53.110390+0000","last_became_peered":"2023-10-30T12:11:53.110390+0000","last_unstale":"2023-10-30T12:32:28.646963+0000","last_undegraded":"2023-10-30T12:32:28.646963+0000","last_fullsized":"2023-10-30T12:32:28.646963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:52.930583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":870,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.986414+0000","last_change":"2023-10-30T12:12:00.804118+0000","last_active":"2023-10-30T12:32:33.986414+0000","last_peered":"2023-10-30T12:32:33.986414+0000","last_clean":"2023-10-30T12:32:33.986414+0000","last_became_active":"2023-10-30T12:12:00.784060+0000","last_became_peered":"2023-10-30T12:12:00.784060+0000","last_unstale":"2023-10-30T12:32:33.986414+0000","last_undegraded":"2023-10-30T12:32:33.986414+0000","last_fullsized":"2023-10-30T12:32:33.986414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":870,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.983490+0000","last_change":"2023-10-30T12:12:01.684690+0000","last_active":"2023-10-30T12:32:33.983490+0000","last_peered":"2023-10-30T12:32:33.983490+0000","last_clean":"2023-10-30T12:32:33.983490+0000","last_became_active":"2023-10-30T12:12:01.679359+0000","last_became_peered":"2023-10-30T12:12:01.679359+0000","last_unstale":"2023-10-30T12:32:33.983490+0000","last_undegraded":"2023-10-30T12:32:33.983490+0000","last_fullsized":"2023-10-30T12:32:33.983490+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":869,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.022486+0000","last_change":"2023-10-30T12:12:08.288834+0000","last_active":"2023-10-30T12:32:33.022486+0000","last_peered":"2023-10-30T12:32:33.022486+0000","last_clean":"2023-10-30T12:32:33.022486+0000","last_became_active":"2023-10-30T12:12:08.259922+0000","last_became_peered":"2023-10-30T12:12:08.259922+0000","last_unstale":"2023-10-30T12:32:33.022486+0000","last_undegraded":"2023-10-30T12:32:33.022486+0000","last_fullsized":"2023-10-30T12:32:33.022486+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477009,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78044,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293796,"statfs":{"total":96636764160,"available":96556847104,"internally_reserved":0,"allocated":6770688,"data_stored":3198410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73136584},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:55 2023","interfaces":[{"interface":"back","average":{"1min":10.798999999999999,"5min":17.038,"15min":20.966000000000001},"min":{"1min":6.4589999999999996,"5min":4.1319999999999997,"15min":4.1319999999999997},"max":{"1min":26.298999999999999,"5min":140.75200000000001,"15min":213.76499999999999},"last":13.858000000000001},{"interface":"front","average":{"1min":13.866,"5min":19.09,"15min":24.885999999999999},"min":{"1min":5.6989999999999998,"5min":4.21,"15min":4.1360000000000001},"max":{"1min":26.111999999999998,"5min":142.81200000000001,"15min":265.09100000000001},"last":9.6790000000000003}]},{"osd":1,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":16.192,"5min":19.501000000000001,"15min":21.971},"min":{"1min":6.9690000000000003,"5min":5.3449999999999998,"15min":4.6609999999999996},"max":{"1min":60.024999999999999,"5min":119.535,"15min":164.755},"last":7.9820000000000002},{"interface":"front","average":{"1min":16.702999999999999,"5min":19.878,"15min":25.541},"min":{"1min":7.5990000000000002,"5min":6.6399999999999997,"15min":6.0990000000000002},"max":{"1min":72.597999999999999,"5min":146.874,"15min":260.19099999999997},"last":6.2389999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:31:55 2023","interfaces":[{"interface":"back","average":{"1min":18.702000000000002,"5min":23.050999999999998,"15min":32.445},"min":{"1min":9.2129999999999992,"5min":6.4589999999999996,"15min":6.4589999999999996},"max":{"1min":98.338999999999999,"5min":215.48400000000001,"15min":277.137},"last":8.4570000000000007},{"interface":"front","average":{"1min":17.099,"5min":23.308,"15min":32.445},"min":{"1min":9.9890000000000008,"5min":6.5860000000000003,"15min":6.5860000000000003},"max":{"1min":74.525999999999996,"5min":254.81999999999999,"15min":265.447},"last":10.257}]},{"osd":4,"last update":"Mon Oct 30 12:31:55 2023","interfaces":[{"interface":"back","average":{"1min":14.472,"5min":22.044,"15min":30.475999999999999},"min":{"1min":8.7599999999999998,"5min":8.7599999999999998,"15min":7.8780000000000001},"max":{"1min":23.257999999999999,"5min":139.517,"15min":287.06200000000001},"last":9.0549999999999997},{"interface":"front","average":{"1min":17.981999999999999,"5min":24.353000000000002,"15min":34.198999999999998},"min":{"1min":11.537000000000001,"5min":11.537000000000001,"15min":10.587},"max":{"1min":28.649000000000001,"5min":130.221,"15min":259.279},"last":13.247999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":20.829000000000001,"5min":28.876999999999999,"15min":35.593000000000004},"min":{"1min":12.195,"5min":11.795,"15min":8.2550000000000008},"max":{"1min":74.197999999999993,"5min":324.26600000000002,"15min":324.26600000000002},"last":49.777000000000001},{"interface":"front","average":{"1min":15.435,"5min":19.82,"15min":24.088000000000001},"min":{"1min":7.1639999999999997,"5min":6.6900000000000004,"15min":6.5679999999999996},"max":{"1min":70.387,"5min":251.57900000000001,"15min":251.57900000000001},"last":39.686}]},{"osd":6,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":21.481999999999999,"5min":28.734000000000002,"15min":28.196999999999999},"min":{"1min":12.811999999999999,"5min":10.584,"15min":7.9370000000000003},"max":{"1min":68.438999999999993,"5min":135.005,"15min":218.80600000000001},"last":11.516999999999999},{"interface":"front","average":{"1min":13.765000000000001,"5min":20.492000000000001,"15min":27.355},"min":{"1min":9.7479999999999993,"5min":8.0120000000000005,"15min":6.8170000000000002},"max":{"1min":25.957999999999998,"5min":133.999,"15min":239.80500000000001},"last":14.928000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":22.760999999999999,"5min":33.521999999999998,"15min":40.765000000000001},"min":{"1min":13.420999999999999,"5min":13.420999999999999,"15min":10.132999999999999},"max":{"1min":67.742999999999995,"5min":323.58199999999999,"15min":1077.606},"last":15.827999999999999},{"interface":"front","average":{"1min":15.550000000000001,"5min":26.276,"15min":29.468},"min":{"1min":9.5069999999999997,"5min":9.0050000000000008,"15min":8.5879999999999992},"max":{"1min":21.239000000000001,"5min":253.04599999999999,"15min":253.04599999999999},"last":16.515000000000001}]}]},{"osd":2,"up_from":16,"seq":68719477008,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60196,"kb_used_data":6616,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311644,"statfs":{"total":96636764160,"available":96575123456,"internally_reserved":0,"allocated":6774784,"data_stored":3198415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":22.933,"5min":16.960999999999999,"15min":24.920000000000002},"min":{"1min":7.641,"5min":4.5149999999999997,"15min":4.5149999999999997},"max":{"1min":205.785,"5min":205.785,"15min":398.01499999999999},"last":6.21},{"interface":"front","average":{"1min":19.140000000000001,"5min":18.173999999999999,"15min":24.411000000000001},"min":{"1min":6.8419999999999996,"5min":5.9969999999999999,"15min":4.4790000000000001},"max":{"1min":135.32499999999999,"5min":135.32499999999999,"15min":376.49299999999999},"last":8.3810000000000002}]},{"osd":1,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":23.818000000000001,"5min":19.024000000000001,"15min":26.135000000000002},"min":{"1min":7.2519999999999998,"5min":7.2519999999999998,"15min":6.9660000000000002},"max":{"1min":172.09999999999999,"5min":172.09999999999999,"15min":317.92099999999999},"last":11.766999999999999},{"interface":"front","average":{"1min":28.353999999999999,"5min":23.835000000000001,"15min":32.107999999999997},"min":{"1min":10.314,"5min":9.7509999999999994,"15min":7.9989999999999997},"max":{"1min":183.803,"5min":183.803,"15min":355.99400000000003},"last":12.417999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":29.100999999999999,"5min":23.119,"15min":29.891999999999999},"min":{"1min":11.067,"5min":9.3870000000000005,"15min":6.4080000000000004},"max":{"1min":207.35300000000001,"5min":207.35300000000001,"15min":412.42899999999997},"last":8.7219999999999995},{"interface":"front","average":{"1min":20.431000000000001,"5min":17.634,"15min":28.106999999999999},"min":{"1min":6.1959999999999997,"5min":6.1959999999999997,"15min":6.1959999999999997},"max":{"1min":147.63900000000001,"5min":147.63900000000001,"15min":328.90899999999999},"last":15.050000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":17.556999999999999,"5min":16.536999999999999,"15min":22.847000000000001},"min":{"1min":8.3580000000000005,"5min":7.556,"15min":6.5439999999999996},"max":{"1min":44.722000000000001,"5min":69.641000000000005,"15min":326.565},"last":11.058},{"interface":"front","average":{"1min":20.829000000000001,"5min":20.02,"15min":27.216999999999999},"min":{"1min":12.151,"5min":10.289,"15min":9.8710000000000004},"max":{"1min":49.927,"5min":83.744,"15min":302.76499999999999},"last":29.786000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":23.760999999999999,"5min":19.529,"15min":22.626000000000001},"min":{"1min":11.141,"5min":5.4139999999999997,"15min":4.9539999999999997},"max":{"1min":91.162000000000006,"5min":112.432,"15min":246.011},"last":10.430999999999999},{"interface":"front","average":{"1min":13.92,"5min":17.667999999999999,"15min":30.521999999999998},"min":{"1min":7.008,"5min":6.492,"15min":6.492},"max":{"1min":89.206000000000003,"5min":155.58699999999999,"15min":350.80399999999997},"last":9.6920000000000002}]},{"osd":6,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":20.018000000000001,"5min":21.427,"15min":29.437999999999999},"min":{"1min":10.337999999999999,"5min":9.5139999999999993,"15min":7.8949999999999996},"max":{"1min":47.087000000000003,"5min":94.465000000000003,"15min":374.16699999999997},"last":27.945},{"interface":"front","average":{"1min":26.401,"5min":24.434000000000001,"15min":31.552},"min":{"1min":15.363,"5min":10.032,"15min":8.2400000000000002},"max":{"1min":82.998000000000005,"5min":98.968999999999994,"15min":333.37299999999999},"last":9.391}]},{"osd":7,"last update":"Mon Oct 30 12:31:49 2023","interfaces":[{"interface":"back","average":{"1min":21.491,"5min":15.154,"15min":23.888000000000002},"min":{"1min":8.4600000000000009,"5min":6.8049999999999997,"15min":6.1539999999999999},"max":{"1min":88.486999999999995,"5min":88.486999999999995,"15min":298.68900000000002},"last":15.698},{"interface":"front","average":{"1min":26.132000000000001,"5min":19.689,"15min":32.052999999999997},"min":{"1min":9.7449999999999992,"5min":9.7449999999999992,"15min":7.4480000000000004},"max":{"1min":114.895,"5min":114.895,"15min":453.28399999999999},"last":32.222000000000001}]}]},{"osd":0,"up_from":16,"seq":68719477008,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59996,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311844,"statfs":{"total":96636764160,"available":96575328256,"internally_reserved":0,"allocated":6770688,"data_stored":3198410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":12.099,"5min":18.317,"15min":20.573},"min":{"1min":6.7759999999999998,"5min":5.6520000000000001,"15min":4.79},"max":{"1min":29.474,"5min":152.46799999999999,"15min":212.69200000000001},"last":7.4240000000000004},{"interface":"front","average":{"1min":11.606999999999999,"5min":17.617000000000001,"15min":20.478000000000002},"min":{"1min":5.1799999999999997,"5min":5.1799999999999997,"15min":5.1799999999999997},"max":{"1min":27.994,"5min":150.90100000000001,"15min":219.46600000000001},"last":10.696999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":16.555,"5min":17.678000000000001,"15min":21.452999999999999},"min":{"1min":7.8090000000000002,"5min":5.9580000000000002,"15min":5.9580000000000002},"max":{"1min":80.406999999999996,"5min":120.355,"15min":294.21699999999998},"last":13.914999999999999},{"interface":"front","average":{"1min":17.379999999999999,"5min":17.045000000000002,"15min":19.501000000000001},"min":{"1min":8.4209999999999994,"5min":6.5499999999999998,"15min":5.8010000000000002},"max":{"1min":96.001999999999995,"5min":101.89100000000001,"15min":237.929},"last":12.952}]},{"osd":3,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":12.368,"5min":20.166,"15min":19.789000000000001},"min":{"1min":5.484,"5min":5.484,"15min":5.484},"max":{"1min":68.653000000000006,"5min":210.88900000000001,"15min":210.88900000000001},"last":11.772},{"interface":"front","average":{"1min":17.181000000000001,"5min":17.488,"15min":21.372},"min":{"1min":7.4690000000000003,"5min":6.2770000000000001,"15min":5.8129999999999997},"max":{"1min":72.388999999999996,"5min":220.589,"15min":220.589},"last":19.588000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":20.411999999999999,"5min":23.530999999999999,"15min":21.559000000000001},"min":{"1min":10.009,"5min":8.8040000000000003,"15min":6.4329999999999998},"max":{"1min":110.872,"5min":247.57300000000001,"15min":247.57300000000001},"last":17.356000000000002},{"interface":"front","average":{"1min":13.563000000000001,"5min":19.027999999999999,"15min":21.359000000000002},"min":{"1min":7.7889999999999997,"5min":6.7370000000000001,"15min":6.7370000000000001},"max":{"1min":23.960999999999999,"5min":267.23899999999998,"15min":267.23899999999998},"last":14.922000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":13.832000000000001,"5min":21.916,"15min":21.706},"min":{"1min":7.3899999999999997,"5min":7.3899999999999997,"15min":6.7080000000000002},"max":{"1min":25.125,"5min":232.041,"15min":232.041},"last":16.254999999999999},{"interface":"front","average":{"1min":16.873000000000001,"5min":25.713000000000001,"15min":25.507000000000001},"min":{"1min":11.106,"5min":9.843,"15min":9.843},"max":{"1min":27.648,"5min":256.58699999999999,"15min":256.58699999999999},"last":20.579999999999998}]},{"osd":6,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":21.965,"5min":23.777999999999999,"15min":24.338999999999999},"min":{"1min":11.445,"5min":8.2750000000000004,"15min":8.2750000000000004},"max":{"1min":116.67100000000001,"5min":136.44399999999999,"15min":235.15199999999999},"last":21.620999999999999},{"interface":"front","average":{"1min":18.707000000000001,"5min":24.120999999999999,"15min":24.725000000000001},"min":{"1min":9.9090000000000007,"5min":8.4619999999999997,"15min":8.4619999999999997},"max":{"1min":56.292000000000002,"5min":196.33799999999999,"15min":196.33799999999999},"last":23.341999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":14.186,"5min":17.329000000000001,"15min":20.686},"min":{"1min":7.851,"5min":7.2110000000000003,"15min":7.2110000000000003},"max":{"1min":20.734000000000002,"5min":95.358999999999995,"15min":232.815},"last":18.138000000000002},{"interface":"front","average":{"1min":19.161000000000001,"5min":22.934000000000001,"15min":26.524000000000001},"min":{"1min":10.509,"5min":9.1110000000000007,"15min":8.827},"max":{"1min":28.800999999999998,"5min":104.599,"15min":199.85300000000001},"last":24.550000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542417,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60192,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311648,"statfs":{"total":96636764160,"available":96575127552,"internally_reserved":0,"allocated":6770688,"data_stored":3198410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":9.0199999999999996,"5min":13.888999999999999,"15min":21.597000000000001},"min":{"1min":4.7370000000000001,"5min":4.1820000000000004,"15min":3.7999999999999998},"max":{"1min":16.635999999999999,"5min":175.166,"15min":410.46699999999998},"last":7.7240000000000002},{"interface":"front","average":{"1min":9.7629999999999999,"5min":14.135999999999999,"15min":23.727},"min":{"1min":4.335,"5min":4.0570000000000004,"15min":4.0570000000000004},"max":{"1min":18.619,"5min":182.875,"15min":1113.894},"last":5.5220000000000002}]},{"osd":1,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":12.548,"5min":14.526,"15min":22.489000000000001},"min":{"1min":5.8879999999999999,"5min":5.1840000000000002,"15min":4.5839999999999996},"max":{"1min":38.268999999999998,"5min":71.530000000000001,"15min":484.42899999999997},"last":11.470000000000001},{"interface":"front","average":{"1min":11.441000000000001,"5min":16.390000000000001,"15min":24.693999999999999},"min":{"1min":7.0030000000000001,"5min":5.1369999999999996,"15min":4.7300000000000004},"max":{"1min":25.640999999999998,"5min":105.95399999999999,"15min":344.97699999999998},"last":9.2249999999999996}]},{"osd":2,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":13.994,"5min":19.303999999999998,"15min":26.847000000000001},"min":{"1min":7.8949999999999996,"5min":7.2370000000000001,"15min":5.242},"max":{"1min":23.893000000000001,"5min":179.96600000000001,"15min":443.49000000000001},"last":14.446999999999999},{"interface":"front","average":{"1min":15.140000000000001,"5min":20.611000000000001,"15min":28.088000000000001},"min":{"1min":7.6820000000000004,"5min":5.883,"15min":5.883},"max":{"1min":30.905000000000001,"5min":186.96899999999999,"15min":318.04000000000002},"last":12.978999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":12.912000000000001,"5min":22.318000000000001,"15min":28.292000000000002},"min":{"1min":7.7729999999999997,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":19.795000000000002,"5min":237.43899999999999,"15min":419.54599999999999},"last":10.006},{"interface":"front","average":{"1min":16.077999999999999,"5min":24.021000000000001,"15min":30.792999999999999},"min":{"1min":8.3559999999999999,"5min":8.0380000000000003,"15min":7.4969999999999999},"max":{"1min":43.798000000000002,"5min":280.57400000000001,"15min":353.90800000000002},"last":16.491}]},{"osd":4,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":20.606000000000002,"5min":19.699000000000002,"15min":25.931000000000001},"min":{"1min":8.0239999999999991,"5min":6.1529999999999996,"15min":5.7039999999999997},"max":{"1min":125.714,"5min":185.97200000000001,"15min":420.41399999999999},"last":15.436},{"interface":"front","average":{"1min":21.055,"5min":21.521000000000001,"15min":27.934000000000001},"min":{"1min":7.1020000000000003,"5min":6.6799999999999997,"15min":6.6799999999999997},"max":{"1min":120.68000000000001,"5min":214.81,"15min":329.49000000000001},"last":17.399999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":14.662000000000001,"5min":19.605,"15min":28.992000000000001},"min":{"1min":8.8040000000000003,"5min":8.4689999999999994,"15min":7.6449999999999996},"max":{"1min":27.105,"5min":172.79300000000001,"15min":449.233},"last":18.103000000000002},{"interface":"front","average":{"1min":20.103000000000002,"5min":24.542999999999999,"15min":33.482999999999997},"min":{"1min":11.194000000000001,"5min":11.194000000000001,"15min":10.382},"max":{"1min":47.503999999999998,"5min":173.39699999999999,"15min":347.05099999999999},"last":19.251000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:31:50 2023","interfaces":[{"interface":"back","average":{"1min":15.154999999999999,"5min":21.318000000000001,"15min":27.535},"min":{"1min":8.4199999999999999,"5min":8.2230000000000008,"15min":6.6059999999999999},"max":{"1min":19.495000000000001,"5min":240.779,"15min":454.25400000000002},"last":12.215},{"interface":"front","average":{"1min":19.113,"5min":24.262,"15min":29.777000000000001},"min":{"1min":12.965999999999999,"5min":9.8710000000000004,"15min":9.8710000000000004},"max":{"1min":42.076000000000001,"5min":198.68199999999999,"15min":507.48700000000002},"last":20.343}]}]},{"osd":1,"up_from":14,"seq":60129542415,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100424,"kb_used_data":6652,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94271416,"statfs":{"total":96636764160,"available":96533929984,"internally_reserved":0,"allocated":6811648,"data_stored":3596255,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":95812038},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":14.826000000000001,"5min":15.159000000000001,"15min":21.579999999999998},"min":{"1min":5.1609999999999996,"5min":5.1609999999999996,"15min":4.1980000000000004},"max":{"1min":64.741,"5min":92.221999999999994,"15min":431.12},"last":10.744999999999999},{"interface":"front","average":{"1min":17.555,"5min":15.711,"15min":21.766999999999999},"min":{"1min":6.2409999999999997,"5min":5.1420000000000003,"15min":4.7199999999999998},"max":{"1min":98.355999999999995,"5min":98.355999999999995,"15min":399.90300000000002},"last":11.901999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":26.707000000000001,"5min":23.045999999999999,"15min":27.221},"min":{"1min":6.7359999999999998,"5min":6.7359999999999998,"15min":6.3339999999999996},"max":{"1min":162.07900000000001,"5min":162.07900000000001,"15min":284.32900000000001},"last":14.053000000000001},{"interface":"front","average":{"1min":21.553000000000001,"5min":19.859000000000002,"15min":25.832000000000001},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":5.8230000000000004},"max":{"1min":118.956,"5min":118.956,"15min":331.928},"last":7.8250000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":23.033999999999999,"5min":24.411000000000001,"15min":31.416},"min":{"1min":8.1790000000000003,"5min":7.1970000000000001,"15min":6.0229999999999997},"max":{"1min":104.30500000000001,"5min":153.827,"15min":270.78100000000001},"last":15.968999999999999},{"interface":"front","average":{"1min":21.942,"5min":24.722000000000001,"15min":29.132000000000001},"min":{"1min":8.5429999999999993,"5min":6.633,"15min":6.4960000000000004},"max":{"1min":106.015,"5min":235.411,"15min":321.42700000000002},"last":16.782}]},{"osd":4,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":22.050000000000001,"5min":26.515000000000001,"15min":27.753},"min":{"1min":10.422000000000001,"5min":8.4659999999999993,"15min":6.6890000000000001},"max":{"1min":143.80600000000001,"5min":196.18000000000001,"15min":278.36399999999998},"last":10.02},{"interface":"front","average":{"1min":21.295999999999999,"5min":24.879999999999999,"15min":25.135999999999999},"min":{"1min":8.8339999999999996,"5min":7.641,"15min":6.3280000000000003},"max":{"1min":148.40100000000001,"5min":219.45400000000001,"15min":284.76900000000001},"last":12.584}]},{"osd":5,"last update":"Mon Oct 30 12:31:46 2023","interfaces":[{"interface":"back","average":{"1min":21.213999999999999,"5min":21.710999999999999,"15min":26.812999999999999},"min":{"1min":7.077,"5min":6.3140000000000001,"15min":6.0869999999999997},"max":{"1min":117.886,"5min":127.604,"15min":346.97300000000001},"last":8.6259999999999994},{"interface":"front","average":{"1min":22.224,"5min":26.302,"15min":29.582000000000001},"min":{"1min":10.473000000000001,"5min":9.8249999999999993,"15min":8.2370000000000001},"max":{"1min":110.979,"5min":280.072,"15min":313.72899999999998},"last":19.817}]},{"osd":6,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":21.413,"5min":24.916,"15min":30.504999999999999},"min":{"1min":8.9939999999999998,"5min":6.7089999999999996,"15min":6.7089999999999996},"max":{"1min":126.279,"5min":126.279,"15min":391.726},"last":17.814},{"interface":"front","average":{"1min":25.158999999999999,"5min":24.603000000000002,"15min":27.167000000000002},"min":{"1min":8.7750000000000004,"5min":8.2289999999999992,"15min":8.0549999999999997},"max":{"1min":149.857,"5min":187.96799999999999,"15min":288.44099999999997},"last":14.803000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":25.628,"5min":25.503,"15min":39.037999999999997},"min":{"1min":12.448,"5min":8.6899999999999995,"15min":8.6899999999999995},"max":{"1min":137.25299999999999,"5min":137.25299999999999,"15min":1115.931},"last":18.975999999999999},{"interface":"front","average":{"1min":29.222999999999999,"5min":32.040999999999997,"15min":39.862000000000002},"min":{"1min":13.659000000000001,"5min":10.536,"15min":9.5079999999999991},"max":{"1min":151.57300000000001,"5min":199.08099999999999,"15min":1115.2660000000001},"last":20.475999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575119,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60192,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311648,"statfs":{"total":96636764160,"available":96575127552,"internally_reserved":0,"allocated":6770688,"data_stored":3198410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:58 2023","interfaces":[{"interface":"back","average":{"1min":9.2989999999999995,"5min":14.662000000000001,"15min":15.426},"min":{"1min":4.0119999999999996,"5min":3.9380000000000002,"15min":3.9380000000000002},"max":{"1min":36.322000000000003,"5min":200.376,"15min":220.25700000000001},"last":8.8350000000000009},{"interface":"front","average":{"1min":9.7750000000000004,"5min":16.427,"15min":15.846},"min":{"1min":5.4180000000000001,"5min":4.6449999999999996,"15min":4.1799999999999997},"max":{"1min":28.044,"5min":196.261,"15min":219.40000000000001},"last":5.9349999999999996}]},{"osd":1,"last update":"Mon Oct 30 12:31:51 2023","interfaces":[{"interface":"back","average":{"1min":13.768000000000001,"5min":18.015000000000001,"15min":18.302},"min":{"1min":7.6390000000000002,"5min":5.1180000000000003,"15min":4.306},"max":{"1min":44.209000000000003,"5min":215.86699999999999,"15min":216.90299999999999},"last":15.451000000000001},{"interface":"front","average":{"1min":15.76,"5min":19.614999999999998,"15min":18.779},"min":{"1min":7.0019999999999998,"5min":5.2039999999999997,"15min":4.5570000000000004},"max":{"1min":42.402000000000001,"5min":225.39699999999999,"15min":225.39699999999999},"last":16.149000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:32:07 2023","interfaces":[{"interface":"back","average":{"1min":12.401,"5min":18.358000000000001,"15min":19.98},"min":{"1min":6.3170000000000002,"5min":5.4729999999999999,"15min":5.4729999999999999},"max":{"1min":93.713999999999999,"5min":245.411,"15min":245.411},"last":9.8759999999999994},{"interface":"front","average":{"1min":14.782999999999999,"5min":22.184000000000001,"15min":22.800000000000001},"min":{"1min":8.9339999999999993,"5min":6.7679999999999998,"15min":6.7679999999999998},"max":{"1min":26.739999999999998,"5min":244.76400000000001,"15min":244.76400000000001},"last":14.804}]},{"osd":3,"last update":"Mon Oct 30 12:31:58 2023","interfaces":[{"interface":"back","average":{"1min":14.455,"5min":25.152000000000001,"15min":22.244},"min":{"1min":7.1970000000000001,"5min":5.9649999999999999,"15min":5.9649999999999999},"max":{"1min":35.399999999999999,"5min":217.495,"15min":300.44600000000003},"last":7.1559999999999997},{"interface":"front","average":{"1min":17.216999999999999,"5min":26.016999999999999,"15min":23.670000000000002},"min":{"1min":10.24,"5min":9.6560000000000006,"15min":8.4960000000000004},"max":{"1min":31.969000000000001,"5min":249.18100000000001,"15min":249.18100000000001},"last":16.539000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:31:51 2023","interfaces":[{"interface":"back","average":{"1min":15.673,"5min":21.34,"15min":23.218},"min":{"1min":9.9610000000000003,"5min":8.75,"15min":6.556},"max":{"1min":34.320999999999998,"5min":246.57599999999999,"15min":246.57599999999999},"last":10.164},{"interface":"front","average":{"1min":21.498000000000001,"5min":20.198,"15min":19.931000000000001},"min":{"1min":7.4119999999999999,"5min":6.9870000000000001,"15min":6.5209999999999999},"max":{"1min":78.716999999999999,"5min":233.952,"15min":233.952},"last":14.467000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:32:34 2023","interfaces":[{"interface":"back","average":{"1min":29.417999999999999,"5min":24.599,"15min":23.084},"min":{"1min":10,"5min":9.3940000000000001,"15min":7.4080000000000004},"max":{"1min":149.303,"5min":267.37799999999999,"15min":267.37799999999999},"last":12.911},{"interface":"front","average":{"1min":26.631,"5min":21.341000000000001,"15min":21.408999999999999},"min":{"1min":8.3149999999999995,"5min":7.0170000000000003,"15min":5.9089999999999998},"max":{"1min":139.23400000000001,"5min":251.63499999999999,"15min":251.63499999999999},"last":10.869999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:32:34 2023","interfaces":[{"interface":"back","average":{"1min":31.233000000000001,"5min":25.446000000000002,"15min":23.885999999999999},"min":{"1min":11.182,"5min":9.8510000000000009,"15min":5.7359999999999998},"max":{"1min":142.589,"5min":240.048,"15min":240.048},"last":11.444000000000001},{"interface":"front","average":{"1min":24.672000000000001,"5min":24.073,"15min":22.902999999999999},"min":{"1min":9.9990000000000006,"5min":7.7110000000000003,"15min":7.4089999999999998},"max":{"1min":166.40100000000001,"5min":250.70400000000001,"15min":250.70400000000001},"last":17.968}]}]},{"osd":5,"up_from":13,"seq":55834575119,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78276,"kb_used_data":6648,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293564,"statfs":{"total":96636764160,"available":96556609536,"internally_reserved":0,"allocated":6807552,"data_stored":3596250,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":13.532999999999999,"5min":15.859999999999999,"15min":16.879999999999999},"min":{"1min":4.6799999999999997,"5min":4.0190000000000001,"15min":3.9620000000000002},"max":{"1min":54.750999999999998,"5min":128.709,"15min":439.73200000000003},"last":24.713000000000001},{"interface":"front","average":{"1min":12.602,"5min":15.343999999999999,"15min":15.458},"min":{"1min":6.1959999999999997,"5min":5.1280000000000001,"15min":4.3170000000000002},"max":{"1min":54.161000000000001,"5min":128,"15min":180.74700000000001},"last":25.486000000000001}]},{"osd":1,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":20.91,"5min":21.420000000000002,"15min":20.076000000000001},"min":{"1min":5.9880000000000004,"5min":5.8789999999999996,"15min":5.0410000000000004},"max":{"1min":73.456999999999994,"5min":213.06700000000001,"15min":235.05199999999999},"last":18.492000000000001},{"interface":"front","average":{"1min":19.178999999999998,"5min":20.559999999999999,"15min":19.687999999999999},"min":{"1min":7.2389999999999999,"5min":5.726,"15min":5.2210000000000001},"max":{"1min":84.025000000000006,"5min":187.249,"15min":420.82900000000001},"last":30.116}]},{"osd":2,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":16.927,"5min":17.652999999999999,"15min":20.605},"min":{"1min":7.4199999999999999,"5min":6.4809999999999999,"15min":5.9539999999999997},"max":{"1min":61.972000000000001,"5min":76.231999999999999,"15min":219.636},"last":32.095999999999997},{"interface":"front","average":{"1min":11.673,"5min":14.134,"15min":18.402000000000001},"min":{"1min":5.3739999999999997,"5min":4.7450000000000001,"15min":4.7450000000000001},"max":{"1min":55.536000000000001,"5min":104.125,"15min":235.387},"last":31.395}]},{"osd":3,"last update":"Mon Oct 30 12:31:59 2023","interfaces":[{"interface":"back","average":{"1min":13.568,"5min":17.634,"15min":21.957999999999998},"min":{"1min":6.3159999999999998,"5min":6.2089999999999996,"15min":5.0739999999999998},"max":{"1min":52.534999999999997,"5min":110.496,"15min":234.13999999999999},"last":24.126000000000001},{"interface":"front","average":{"1min":17.343,"5min":19.657,"15min":24.550999999999998},"min":{"1min":8.4339999999999993,"5min":6.8070000000000004,"15min":6.5330000000000004},"max":{"1min":58.048000000000002,"5min":135.05500000000001,"15min":232.31700000000001},"last":34.457000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":17.914000000000001,"5min":19.361999999999998,"15min":21.667999999999999},"min":{"1min":7.6159999999999997,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":59.853999999999999,"5min":200.67599999999999,"15min":265.50400000000002},"last":22.390000000000001},{"interface":"front","average":{"1min":17.628,"5min":18.516999999999999,"15min":21.404},"min":{"1min":6.3600000000000003,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":58.649000000000001,"5min":108.554,"15min":280.72000000000003},"last":37.116999999999997}]},{"osd":6,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":21.899999999999999,"5min":20.135000000000002,"15min":23.164000000000001},"min":{"1min":11.425000000000001,"5min":8.1229999999999993,"15min":8.1229999999999993},"max":{"1min":76.626999999999995,"5min":86.918000000000006,"15min":421.87599999999998},"last":30.75},{"interface":"front","average":{"1min":20,"5min":22.122,"15min":23.684000000000001},"min":{"1min":8.8409999999999993,"5min":7.3460000000000001,"15min":7.3460000000000001},"max":{"1min":61.104999999999997,"5min":182.376,"15min":353.63099999999997},"last":36.341999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:31:54 2023","interfaces":[{"interface":"back","average":{"1min":21.384,"5min":23.309000000000001,"15min":25.530000000000001},"min":{"1min":12.462999999999999,"5min":9.4359999999999999,"15min":6.96},"max":{"1min":59.232999999999997,"5min":124.621,"15min":854.64300000000003},"last":22.971},{"interface":"front","average":{"1min":13.034000000000001,"5min":17.204000000000001,"15min":19.34},"min":{"1min":7.5110000000000001,"5min":6.2670000000000003,"15min":5.5300000000000002},"max":{"1min":23.940000000000001,"5min":107.976,"15min":264.84699999999998},"last":35.280999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575119,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82400,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289440,"statfs":{"total":96636764160,"available":96552386560,"internally_reserved":0,"allocated":6770688,"data_stored":3198410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":11.680999999999999,"5min":17.094000000000001,"15min":21.251999999999999},"min":{"1min":6.5650000000000004,"5min":5.1920000000000002,"15min":3.8650000000000002},"max":{"1min":35.334000000000003,"5min":166.20400000000001,"15min":372.52300000000002},"last":7.6310000000000002},{"interface":"front","average":{"1min":14.744,"5min":20.887,"15min":24.428999999999998},"min":{"1min":8.5109999999999992,"5min":6.0110000000000001,"15min":4.9660000000000002},"max":{"1min":30.943000000000001,"5min":167.251,"15min":200.37299999999999},"last":13.369}]},{"osd":1,"last update":"Mon Oct 30 12:32:32 2023","interfaces":[{"interface":"back","average":{"1min":24.058,"5min":20.231000000000002,"15min":22.433},"min":{"1min":5.5039999999999996,"5min":5.5039999999999996,"15min":5.5039999999999996},"max":{"1min":126.119,"5min":167.84700000000001,"15min":231.49799999999999},"last":10.021000000000001},{"interface":"front","average":{"1min":20.042999999999999,"5min":16.888000000000002,"15min":19.818999999999999},"min":{"1min":5.79,"5min":5.2690000000000001,"15min":4.641},"max":{"1min":112.584,"5min":223.441,"15min":223.441},"last":9.4550000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:32:32 2023","interfaces":[{"interface":"back","average":{"1min":15.933999999999999,"5min":23.056000000000001,"15min":26.792000000000002},"min":{"1min":6.2549999999999999,"5min":6.2549999999999999,"15min":6.2149999999999999},"max":{"1min":55.453000000000003,"5min":198.66999999999999,"15min":438.15499999999997},"last":14.56},{"interface":"front","average":{"1min":14.124000000000001,"5min":18.303000000000001,"15min":21.728000000000002},"min":{"1min":6.5380000000000003,"5min":6.2220000000000004,"15min":4.944},"max":{"1min":41.518999999999998,"5min":254.06399999999999,"15min":254.06399999999999},"last":11.385999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:32:04 2023","interfaces":[{"interface":"back","average":{"1min":17.190999999999999,"5min":23.48,"15min":28.890000000000001},"min":{"1min":10.754,"5min":6.9710000000000001,"15min":6.9710000000000001},"max":{"1min":52.381999999999998,"5min":269.06700000000001,"15min":269.06700000000001},"last":11.951000000000001},{"interface":"front","average":{"1min":11.754,"5min":16.59,"15min":22.030999999999999},"min":{"1min":4.8230000000000004,"5min":4.8230000000000004,"15min":4.8230000000000004},"max":{"1min":53.374000000000002,"5min":239.715,"15min":239.715},"last":5.2400000000000002}]},{"osd":5,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":16.629000000000001,"5min":23.850000000000001,"15min":24.184999999999999},"min":{"1min":10.301,"5min":8.0670000000000002,"15min":8.0039999999999996},"max":{"1min":38.497,"5min":268.34199999999998,"15min":268.34199999999998},"last":10.759},{"interface":"front","average":{"1min":15.297000000000001,"5min":20.027000000000001,"15min":21.085000000000001},"min":{"1min":8.8490000000000002,"5min":6.1379999999999999,"15min":5.452},"max":{"1min":36.853000000000002,"5min":252.899,"15min":252.899},"last":16.584}]},{"osd":6,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":17.149000000000001,"5min":20.524999999999999,"15min":24.902000000000001},"min":{"1min":8.9629999999999992,"5min":8.8559999999999999,"15min":7.21},"max":{"1min":51.32,"5min":273.90100000000001,"15min":273.90100000000001},"last":12.646000000000001},{"interface":"front","average":{"1min":8.6199999999999992,"5min":20.821000000000002,"15min":28.239000000000001},"min":{"1min":5.8849999999999998,"5min":5.8849999999999998,"15min":5.8849999999999998},"max":{"1min":13.861000000000001,"5min":292.11099999999999,"15min":292.11099999999999},"last":8.4280000000000008}]},{"osd":7,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":17.405999999999999,"5min":25.300000000000001,"15min":30.454999999999998},"min":{"1min":11.175000000000001,"5min":7.4740000000000002,"15min":6.46},"max":{"1min":38.466000000000001,"5min":239.166,"15min":652.50699999999995},"last":13.997},{"interface":"front","average":{"1min":21.138000000000002,"5min":24.059999999999999,"15min":27.285},"min":{"1min":12.268000000000001,"5min":10.422000000000001,"15min":8.9309999999999992},"max":{"1min":114.399,"5min":303.19999999999999,"15min":303.19999999999999},"last":17.399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:32:40.238 INFO:tasks.ceph:pgid 24.4 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.239 INFO:tasks.ceph:pgid 24.5 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.239 INFO:tasks.ceph:pgid 24.6 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.239 INFO:tasks.ceph:pgid 24.7 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.240 INFO:tasks.ceph:pgid 24.0 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.240 INFO:tasks.ceph:pgid 24.1 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.240 INFO:tasks.ceph:pgid 24.2 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.240 INFO:tasks.ceph:pgid 24.3 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.241 INFO:tasks.ceph:pgid 24.c last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.241 INFO:tasks.ceph:pgid 24.d last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.241 INFO:tasks.ceph:pgid 24.e last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.241 INFO:tasks.ceph:pgid 24.f last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.242 INFO:tasks.ceph:pgid 24.1d last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.242 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-30T12:10:15.050608+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=10, tm_sec=15, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.242 INFO:tasks.ceph:pgid 24.1c last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.243 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-30T12:10:15.050608+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=10, tm_sec=15, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.243 INFO:tasks.ceph:pgid 24.1f last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.243 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T12:10:15.050608+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=10, tm_sec=15, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.243 INFO:tasks.ceph:pgid 24.18 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.244 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-30T12:10:15.050608+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=10, tm_sec=15, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.244 INFO:tasks.ceph:pgid 24.1b last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.244 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-30T12:10:15.050608+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=10, tm_sec=15, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.244 INFO:tasks.ceph:pgid 24.1a last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.245 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-30T12:10:07.000545+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=10, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.245 INFO:tasks.ceph:pgid 24.13 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.245 INFO:tasks.ceph:pgid 24.12 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.246 INFO:tasks.ceph:pgid 24.11 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.246 INFO:tasks.ceph:pgid 24.10 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.246 INFO:tasks.ceph:pgid 24.17 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.247 INFO:tasks.ceph:pgid 24.16 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.247 INFO:tasks.ceph:pgid 24.15 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.247 INFO:tasks.ceph:pgid 24.14 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.247 INFO:tasks.ceph:pgid 24.b last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.248 INFO:tasks.ceph:pgid 24.a last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.248 INFO:tasks.ceph:pgid 24.9 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.248 INFO:tasks.ceph:pgid 24.8 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:32:40.248 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T12:32:43.010 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:42.945 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:43.011 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:42.946 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:43.022 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:42.958 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:43.056 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:42.965 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:43.057 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:42.966 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:43.069 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:42.977 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:43.073 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:43.008 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:43.076 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:43.012 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:43.089 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:43.025 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.071 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:58.006 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.072 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:58.008 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.088 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:58.023 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.117 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:58.052 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.117 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:58.026 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.121 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:58.056 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.125 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:58.033 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.138 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:58.074 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.141 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:58.049 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.723 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:58.631 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.723 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:58.632 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.732 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:58.640 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.764 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:58.700 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.765 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:58.701 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.767 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:58.702 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.768 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:58.703 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.773 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:23:58.709 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:32:58.776 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:23:58.711 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:00.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:33:00.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.373+0000 7f90d102d640 10 monclient: get_monmap_and_config 2023-10-30T12:33:00.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.373+0000 7f90d102d640 10 monclient: build_initial_monmap 2023-10-30T12:33:00.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.373+0000 7f90d102d640 10 monclient: monmap: 2023-10-30T12:33:00.375 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:33:00.375 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:33:00.376 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:33:00.374158+0000 2023-10-30T12:33:00.376 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:33:00.374158+0000 2023-10-30T12:33:00.376 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:33:00.376 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:33:00.376 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:33:00.377 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:33:00.377 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:33:00.377 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:33:00.377 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:33:00.377 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:00.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.373+0000 7f90d102d640 1 Processor -- start 2023-10-30T12:33:00.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.373+0000 7f90d102d640 1 -- start start 2023-10-30T12:33:00.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.373+0000 7f90d102d640 10 monclient: init 2023-10-30T12:33:00.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.374+0000 7f90d102d640 10 monclient: _reopen_session rank -1 2023-10-30T12:33:00.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.374+0000 7f90d102d640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-10-30T12:33:00.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.374+0000 7f90d102d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f90cc0aea90 0x7f90cc141900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:00.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.374+0000 7f90d102d640 10 monclient(hunting): picked mon.noname-a con 0x7f90cc0aea90 addr v2:172.21.15.3:3300/0 2023-10-30T12:33:00.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.374+0000 7f90d102d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f90cc0b12c0 addr v1:172.21.15.3:6789/0 2023-10-30T12:33:00.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.374+0000 7f90d102d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f90cc141e40 addr v1:172.21.15.92:6789/0 2023-10-30T12:33:00.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.374+0000 7f90d102d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc0b0e20 con 0x7f90cc0b12c0 2023-10-30T12:33:00.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.374+0000 7f90d102d640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc0afe80 con 0x7f90cc141e40 2023-10-30T12:33:00.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.374+0000 7f90d102d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:33:00.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.374+0000 7f90d102d640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90cc0679a0 con 0x7f90cc0aea90 2023-10-30T12:33:00.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.374+0000 7f90d102d640 10 monclient(hunting): _renew_subs 2023-10-30T12:33:00.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.374+0000 7f90d102d640 10 monclient(hunting): authenticate will time out at 2771.588580s 2023-10-30T12:33:00.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.375+0000 7f90cad9d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f90cc0aea90 0x7f90cc141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:00.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.375+0000 7f90cb59e640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f90cc141e40 0x7f90cc14c210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.3:32864/0 (socket says 172.21.15.3:32864) 2023-10-30T12:33:00.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.375+0000 7f90cb59e640 1 -- 172.21.15.3:0/3986548738 learned_addr learned my addr 172.21.15.3:0/3986548738 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:33:00.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.375+0000 7f90cad9d640 10 monclient(hunting): get_auth_request con 0x7f90cc0aea90 auth_method 0 2023-10-30T12:33:00.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.375+0000 7f90cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:33:00.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.375+0000 7f90cad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:33:00.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.375+0000 7f90cad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:33:00.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.376+0000 7f90cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:33:00.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.376+0000 7f90cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:33:00.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.376+0000 7f90cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:33:00.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.378+0000 7f90cad9d640 10 monclient(hunting): handle_auth_done global_id 16701 payload 274 2023-10-30T12:33:00.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.378+0000 7f90cad9d640 1 -- 172.21.15.3:0/3986548738 >> v1:172.21.15.92:6789/0 conn(0x7f90cc141e40 legacy=0x7f90cc14c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:00.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.378+0000 7f90cad9d640 1 -- 172.21.15.3:0/3986548738 >> v1:172.21.15.3:6789/0 conn(0x7f90cc0b12c0 legacy=0x7f90cc0ad520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:00.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.378+0000 7f90cad9d640 10 monclient: _finish_hunting 0 2023-10-30T12:33:00.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.378+0000 7f90cad9d640 1 monclient: found mon.noname-a 2023-10-30T12:33:00.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.378+0000 7f90cad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:33:00.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.378+0000 7f90cad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.3:3300/0 2023-10-30T12:33:00.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.378+0000 7f90cad9d640 1 -- 172.21.15.3:0/3986548738 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc0b0160 con 0x7f90cc0aea90 2023-10-30T12:33:00.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.383+0000 7f90cad9d640 1 --2- 172.21.15.3:0/3986548738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc0aea90 0x7f90cc141900 secure :-1 s=READY pgs=3619 cs=0 l=1 rev1=1 crypto rx=0x7f90c40097a0 tx=0x7f90c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622735556fa1967e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:00.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90c9d9b640 1 -- 172.21.15.3:0/3986548738 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4031830 con 0x7f90cc0aea90 2023-10-30T12:33:00.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:33:00.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90c9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:33:00.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90c9d9b640 10 monclient: dump: 2023-10-30T12:33:00.399 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:33:00.399 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:33:00.400 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:00.400 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:00.400 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:33:00.401 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:33:00.402 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:33:00.402 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:33:00.402 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:33:00.402 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:00.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90c9d9b640 10 monclient: _finish_auth 0 2023-10-30T12:33:00.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90c9d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:33:00.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:33:00.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90c9d9b640 1 -- 172.21.15.3:0/3986548738 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c402d040 con 0x7f90cc0aea90 2023-10-30T12:33:00.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90c9d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:33:00.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90c9d9b640 1 -- 172.21.15.3:0/3986548738 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4031bc0 con 0x7f90cc0aea90 2023-10-30T12:33:00.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90d102d640 5 monclient: authenticate success, global_id 16701 2023-10-30T12:33:00.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90d102d640 10 monclient: get_monmap_and_config success 2023-10-30T12:33:00.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90d102d640 4 set_mon_vals no callback set 2023-10-30T12:33:00.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:33:00.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90c9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:33:00.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.389+0000 7f90c9d9b640 10 monclient: dump: 2023-10-30T12:33:00.406 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:33:00.407 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:33:00.407 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:00.407 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:00.408 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:33:00.408 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:33:00.408 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:33:00.409 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:33:00.409 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:33:00.409 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:00.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.390+0000 7f90d102d640 10 monclient: shutdown 2023-10-30T12:33:00.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.390+0000 7f90d102d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:33:00.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.390+0000 7f90d102d640 1 -- 172.21.15.3:0/3986548738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc0aea90 msgr2=0x7f90cc141900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:00.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.390+0000 7f90d102d640 1 --2- 172.21.15.3:0/3986548738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc0aea90 0x7f90cc141900 secure :-1 s=READY pgs=3619 cs=0 l=1 rev1=1 crypto rx=0x7f90c40097a0 tx=0x7f90c402b8d0 comp rx=0 tx=0).stop 2023-10-30T12:33:00.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.390+0000 7f90d102d640 1 -- 172.21.15.3:0/3986548738 shutdown_connections 2023-10-30T12:33:00.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.390+0000 7f90d102d640 1 --2- 172.21.15.3:0/3986548738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc0aea90 0x7f90cc141900 unknown :-1 s=CLOSED pgs=3619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:00.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.390+0000 7f90d102d640 1 -- 172.21.15.3:0/3986548738 >> 172.21.15.3:0/3986548738 conn(0x7f90cc0b7120 msgr2=0x7f90cc0b7500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:00.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.390+0000 7f90d102d640 1 -- 172.21.15.3:0/3986548738 shutdown_connections 2023-10-30T12:33:00.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.390+0000 7f90d102d640 1 -- 172.21.15.3:0/3986548738 wait complete. 2023-10-30T12:33:00.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.391+0000 7f90d102d640 10 monclient: build_initial_monmap 2023-10-30T12:33:00.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.391+0000 7f90d102d640 10 monclient: monmap: 2023-10-30T12:33:00.412 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:33:00.412 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:33:00.412 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:33:00.412 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:33:00.413 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:33:00.413 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:33:00.413 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:33:00.413 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:33:00.413 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:33:00.414 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:00.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.391+0000 7f90d102d640 1 librados: starting msgr at 2023-10-30T12:33:00.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.391+0000 7f90d102d640 1 librados: starting objecter 2023-10-30T12:33:00.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 1 Processor -- start 2023-10-30T12:33:00.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 1 -- start start 2023-10-30T12:33:00.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 1 librados: setting wanted keys 2023-10-30T12:33:00.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 1 librados: calling monclient init 2023-10-30T12:33:00.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 10 monclient: init 2023-10-30T12:33:00.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 10 monclient: _reopen_session rank -1 2023-10-30T12:33:00.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:33:00.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc0aea90 0x7f90cc1449f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:00.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 10 monclient(hunting): picked mon.noname-b con 0x7f90cc0aea90 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:33:00.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f90cc0b12c0 0x7f90cc1d65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:00.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 10 monclient(hunting): picked mon.noname-c con 0x7f90cc0b12c0 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:33:00.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc141e40 0x7f90cc1d6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:00.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 10 monclient(hunting): picked mon.noname-a con 0x7f90cc141e40 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:33:00.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:33:00.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc0b38a0 con 0x7f90cc141e40 2023-10-30T12:33:00.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:33:00.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc1d7040 con 0x7f90cc0aea90 2023-10-30T12:33:00.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:33:00.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc13a520 con 0x7f90cc0b12c0 2023-10-30T12:33:00.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 10 monclient(hunting): _renew_subs 2023-10-30T12:33:00.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.392+0000 7f90d102d640 10 monclient(hunting): authenticate will time out at 2771.606762s 2023-10-30T12:33:00.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90cb59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc141e40 0x7f90cc1d6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:00.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90ca59c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f90cc0b12c0 0x7f90cc1d65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:00.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90cad9d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc0aea90 0x7f90cc1449f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:00.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90cb59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc141e40 0x7f90cc1d6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58346/0 (socket says 172.21.15.3:58346) 2023-10-30T12:33:00.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90cb59e640 1 -- 172.21.15.3:0/619792280 learned_addr learned my addr 172.21.15.3:0/619792280 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:33:00.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90cb59e640 10 monclient(hunting): get_auth_request con 0x7f90cc141e40 auth_method 0 2023-10-30T12:33:00.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90cb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:33:00.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90cb59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:33:00.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90cb59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:33:00.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90ca59c640 10 monclient(hunting): get_auth_request con 0x7f90cc0b12c0 auth_method 0 2023-10-30T12:33:00.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:33:00.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90ca59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:33:00.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90ca59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:33:00.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90cad9d640 10 monclient(hunting): get_auth_request con 0x7f90cc0aea90 auth_method 0 2023-10-30T12:33:00.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:33:00.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90cad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:33:00.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.393+0000 7f90cad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:33:00.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.394+0000 7f90cb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:33:00.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.394+0000 7f90cb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:33:00.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.394+0000 7f90cb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:33:00.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.394+0000 7f90ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:33:00.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.394+0000 7f90ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:33:00.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.394+0000 7f90ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:33:00.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.394+0000 7f90cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:33:00.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.394+0000 7f90cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:33:00.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.394+0000 7f90cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:33:00.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.396+0000 7f90cad9d640 10 monclient(hunting): handle_auth_done global_id 21904 payload 824 2023-10-30T12:33:00.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.396+0000 7f90cad9d640 1 -- 172.21.15.3:0/619792280 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f90cc0b12c0 msgr2=0x7f90cc1d65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:00.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.396+0000 7f90cad9d640 1 --2- 172.21.15.3:0/619792280 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f90cc0b12c0 0x7f90cc1d65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:00.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.396+0000 7f90cad9d640 1 -- 172.21.15.3:0/619792280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc141e40 msgr2=0x7f90cc1d6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:00.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.397+0000 7f90cad9d640 1 --2- 172.21.15.3:0/619792280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc141e40 0x7f90cc1d6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:00.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.397+0000 7f90cad9d640 10 monclient: _finish_hunting 0 2023-10-30T12:33:00.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.397+0000 7f90cad9d640 1 monclient: found mon.noname-b 2023-10-30T12:33:00.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.397+0000 7f90cad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:33:00.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.397+0000 7f90cad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:33:00.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.397+0000 7f90cad9d640 1 -- 172.21.15.3:0/619792280 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc13a6a0 con 0x7f90cc0aea90 2023-10-30T12:33:00.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.397+0000 7f90cb59e640 1 --2- 172.21.15.3:0/619792280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc141e40 0x7f90cc1d6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:33:00.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.399+0000 7f90cad9d640 1 --2- 172.21.15.3:0/619792280 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc0aea90 0x7f90cc1449f0 secure :-1 s=READY pgs=5327 cs=0 l=1 rev1=1 crypto rx=0x7f90c4033000 tx=0x7f90c40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:00.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.399+0000 7f90b37fe640 10 client.?.objecter ms_handle_connect 0x7f90cc0aea90 2023-10-30T12:33:00.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.399+0000 7f90b37fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:33:00.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90b37fe640 1 -- 172.21.15.3:0/619792280 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c403a630 con 0x7f90cc0aea90 2023-10-30T12:33:00.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:33:00.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:33:00.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90b37fe640 10 monclient: dump: 2023-10-30T12:33:00.432 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:33:00.432 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:33:00.432 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:00.433 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:00.433 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:33:00.433 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:33:00.433 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:33:00.433 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:33:00.434 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:33:00.434 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:00.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90b37fe640 10 monclient: _finish_auth 0 2023-10-30T12:33:00.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90b37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:33:00.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:33:00.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90b37fe640 1 -- 172.21.15.3:0/619792280 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c403f050 con 0x7f90cc0aea90 2023-10-30T12:33:00.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90b37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:33:00.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90b37fe640 1 -- 172.21.15.3:0/619792280 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c403aa00 con 0x7f90cc0aea90 2023-10-30T12:33:00.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:33:00.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:33:00.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90b37fe640 10 monclient: dump: 2023-10-30T12:33:00.436 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:33:00.436 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:33:00.436 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:00.437 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:00.437 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:33:00.437 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:33:00.437 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:33:00.437 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:33:00.438 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:33:00.438 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:00.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90d102d640 5 monclient: authenticate success, global_id 21904 2023-10-30T12:33:00.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90d102d640 10 monclient: _renew_subs 2023-10-30T12:33:00.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90d102d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:33:00.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90d102d640 1 -- 172.21.15.3:0/619792280 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90cc13a8a0 con 0x7f90cc0aea90 2023-10-30T12:33:00.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90c959a640 4 set_mon_vals no callback set 2023-10-30T12:33:00.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90d102d640 10 client.21904.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:33:00.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90d102d640 10 monclient: _renew_subs 2023-10-30T12:33:00.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90d102d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:33:00.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90d102d640 1 -- 172.21.15.3:0/619792280 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90cc13ace0 con 0x7f90cc0aea90 2023-10-30T12:33:00.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.402+0000 7f90d102d640 1 librados: init done 2023-10-30T12:33:00.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.403+0000 7f90d102d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:33:00.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.403+0000 7f90d102d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:33:00.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.403+0000 7f90d102d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:33:00.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.403+0000 7f90d102d640 1 -- 172.21.15.3:0/619792280 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90cc04e9b0 con 0x7f90cc0aea90 2023-10-30T12:33:00.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.640+0000 7f90b37fe640 1 -- 172.21.15.3:0/619792280 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f90c404c430 con 0x7f90cc0aea90 2023-10-30T12:33:00.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.640+0000 7f90b37fe640 1 --2- 172.21.15.3:0/619792280 >> v2:172.21.15.92:6800/1750155735 conn(0x7f90b806c840 0x7f90b806ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:00.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.641+0000 7f90ca59c640 1 --2- 172.21.15.3:0/619792280 >> v2:172.21.15.92:6800/1750155735 conn(0x7f90b806c840 0x7f90b806ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:00.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.641+0000 7f90ca59c640 10 monclient: get_auth_request con 0x7f90b806c840 auth_method 0 2023-10-30T12:33:00.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.641+0000 7f90ca59c640 1 --2- 172.21.15.3:0/619792280 >> v2:172.21.15.92:6800/1750155735 conn(0x7f90b806c840 0x7f90b806ed00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f90cc0ae0a0 tx=0x7f90c0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:00.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.641+0000 7f90b37fe640 10 client.21904.objecter ms_handle_connect 0x7f90b806c840 2023-10-30T12:33:00.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.642+0000 7f90b37fe640 1 -- 172.21.15.3:0/619792280 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f90c4079570 con 0x7f90cc0aea90 2023-10-30T12:33:00.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.642+0000 7f90b37fe640 10 client.21904.objecter ms_dispatch 0x7f90cc000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:33:00.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.642+0000 7f90b37fe640 3 client.21904.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:33:00.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.642+0000 7f90b37fe640 3 client.21904.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:33:00.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.643+0000 7f90b37fe640 20 client.21904.objecter dump_active .. 0 homeless 2023-10-30T12:33:00.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.653+0000 7f90b37fe640 1 -- 172.21.15.3:0/619792280 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f90c407d610 con 0x7f90cc0aea90 2023-10-30T12:33:00.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.653+0000 7f90b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:33:00.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.653+0000 7f90b37fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:33:00.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.828+0000 7f90d102d640 1 -- 172.21.15.3:0/619792280 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f90cc0b0440 con 0x7f90b806c840 2023-10-30T12:33:00.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.833+0000 7f90b37fe640 1 -- 172.21.15.3:0/619792280 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151381 (secure 0 0 0) 0x7f90c00014a0 con 0x7f90b806c840 2023-10-30T12:33:00.836 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:33:00.837 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-10-30T12:33:00.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 10 librados: watch_flush enter 2023-10-30T12:33:00.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 10 librados: watch_flush exit 2023-10-30T12:33:00.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 20 client.21904.objecter shutdown clearing up homeless session... 2023-10-30T12:33:00.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 10 client.21904.objecter successfully canceled tick 2023-10-30T12:33:00.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 1 -- 172.21.15.3:0/619792280 >> v2:172.21.15.92:6800/1750155735 conn(0x7f90b806c840 msgr2=0x7f90b806ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:00.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 1 --2- 172.21.15.3:0/619792280 >> v2:172.21.15.92:6800/1750155735 conn(0x7f90b806c840 0x7f90b806ed00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f90cc0ae0a0 tx=0x7f90c0009040 comp rx=0 tx=0).stop 2023-10-30T12:33:00.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 10 monclient: shutdown 2023-10-30T12:33:00.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:33:00.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 1 -- 172.21.15.3:0/619792280 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc0aea90 msgr2=0x7f90cc1449f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:00.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 1 --2- 172.21.15.3:0/619792280 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc0aea90 0x7f90cc1449f0 secure :-1 s=READY pgs=5327 cs=0 l=1 rev1=1 crypto rx=0x7f90c4033000 tx=0x7f90c40046c0 comp rx=0 tx=0).stop 2023-10-30T12:33:00.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 1 -- 172.21.15.3:0/619792280 shutdown_connections 2023-10-30T12:33:00.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 1 --2- 172.21.15.3:0/619792280 >> v2:172.21.15.92:6800/1750155735 conn(0x7f90b806c840 0x7f90b806ed00 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:00.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 1 --2- 172.21.15.3:0/619792280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc141e40 0x7f90cc1d6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:00.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 1 --2- 172.21.15.3:0/619792280 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f90cc0b12c0 0x7f90cc1d65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:00.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 1 --2- 172.21.15.3:0/619792280 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc0aea90 0x7f90cc1449f0 unknown :-1 s=CLOSED pgs=5327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:00.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 1 -- 172.21.15.3:0/619792280 >> 172.21.15.3:0/619792280 conn(0x7f90cc0b7120 msgr2=0x7f90cc0b7500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:00.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 1 -- 172.21.15.3:0/619792280 shutdown_connections 2023-10-30T12:33:00.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 1 -- 172.21.15.3:0/619792280 wait complete. 2023-10-30T12:33:00.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:00.836+0000 7f90d102d640 1 librados: shutdown 2023-10-30T12:33:00.848 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1593,"stamp":"2023-10-30T12:33:00.112354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":578544,"kb_used_data":52976,"kb_used_omap":12,"kb_used_meta":525491,"kb_avail":754396176,"statfs":{"total":773094113280,"available":772501684224,"internally_reserved":0,"allocated":54247424,"data_stored":26382970,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12743,"internal_metadata":538103353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002515"},"pg_stats":[{"pgid":"24.4","version":"0'0","reported_seq":863,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:58.845151+0000","last_change":"2023-10-30T12:32:58.849370+0000","last_active":"2023-10-30T12:32:58.845151+0000","last_peered":"2023-10-30T12:32:58.845151+0000","last_clean":"2023-10-30T12:32:58.845151+0000","last_became_active":"2023-10-30T12:11:50.811294+0000","last_became_peered":"2023-10-30T12:11:50.811294+0000","last_unstale":"2023-10-30T12:32:58.845151+0000","last_undegraded":"2023-10-30T12:32:58.845151+0000","last_fullsized":"2023-10-30T12:32:58.845151+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:58.843751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:58.843751+0000","last_clean_scrub_stamp":"2023-10-30T12:32:58.843751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:13.308129+0000","scrub_duration":0.031572250000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":894,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:47.592582+0000","last_change":"2023-10-30T12:32:47.592582+0000","last_active":"2023-10-30T12:32:47.592582+0000","last_peered":"2023-10-30T12:32:47.592582+0000","last_clean":"2023-10-30T12:32:47.592582+0000","last_became_active":"2023-10-30T12:11:58.558569+0000","last_became_peered":"2023-10-30T12:11:58.558569+0000","last_unstale":"2023-10-30T12:32:47.592582+0000","last_undegraded":"2023-10-30T12:32:47.592582+0000","last_fullsized":"2023-10-30T12:32:47.592582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:47.590178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:47.590178+0000","last_clean_scrub_stamp":"2023-10-30T12:32:47.590178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:11:38.145895+0000","scrub_duration":0.046353002999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":893,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:48.584463+0000","last_change":"2023-10-30T12:32:48.586650+0000","last_active":"2023-10-30T12:32:48.584463+0000","last_peered":"2023-10-30T12:32:48.584463+0000","last_clean":"2023-10-30T12:32:48.584463+0000","last_became_active":"2023-10-30T12:11:58.495252+0000","last_became_peered":"2023-10-30T12:11:58.495252+0000","last_unstale":"2023-10-30T12:32:48.584463+0000","last_undegraded":"2023-10-30T12:32:48.584463+0000","last_fullsized":"2023-10-30T12:32:48.584463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:48.580750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:48.580750+0000","last_clean_scrub_stamp":"2023-10-30T12:32:48.580750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:38:33.376709+0000","scrub_duration":0.0492234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":869,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.049256+0000","last_change":"2023-10-30T12:12:25.951506+0000","last_active":"2023-10-30T12:32:33.049256+0000","last_peered":"2023-10-30T12:32:33.049256+0000","last_clean":"2023-10-30T12:32:33.049256+0000","last_became_active":"2023-10-30T12:12:25.941872+0000","last_became_peered":"2023-10-30T12:12:25.941872+0000","last_unstale":"2023-10-30T12:32:33.049256+0000","last_undegraded":"2023-10-30T12:32:33.049256+0000","last_fullsized":"2023-10-30T12:32:33.049256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":880,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:43.079860+0000","last_change":"2023-10-30T12:32:43.080900+0000","last_active":"2023-10-30T12:32:43.079860+0000","last_peered":"2023-10-30T12:32:43.079860+0000","last_clean":"2023-10-30T12:32:43.079860+0000","last_became_active":"2023-10-30T12:11:51.581699+0000","last_became_peered":"2023-10-30T12:11:51.581699+0000","last_unstale":"2023-10-30T12:32:43.079860+0000","last_undegraded":"2023-10-30T12:32:43.079860+0000","last_fullsized":"2023-10-30T12:32:43.079860+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:43.078454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:43.078454+0000","last_clean_scrub_stamp":"2023-10-30T12:32:43.078454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:22:47.051220+0000","scrub_duration":0.042524077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1","version":"900'8","reported_seq":878,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:44.206487+0000","last_change":"2023-10-30T12:32:44.207698+0000","last_active":"2023-10-30T12:32:44.206487+0000","last_peered":"2023-10-30T12:32:44.206487+0000","last_clean":"2023-10-30T12:32:44.206487+0000","last_became_active":"2023-10-30T12:12:02.320185+0000","last_became_peered":"2023-10-30T12:12:02.320185+0000","last_unstale":"2023-10-30T12:32:44.206487+0000","last_undegraded":"2023-10-30T12:32:44.206487+0000","last_fullsized":"2023-10-30T12:32:44.206487+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"900'8","last_scrub_stamp":"2023-10-30T12:32:44.205034+0000","last_deep_scrub":"900'8","last_deep_scrub_stamp":"2023-10-30T12:32:44.205034+0000","last_clean_scrub_stamp":"2023-10-30T12:32:44.205034+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:31:51.379361+0000","scrub_duration":0.092655721999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":864,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:45.149747+0000","last_change":"2023-10-30T12:32:45.150862+0000","last_active":"2023-10-30T12:32:45.149747+0000","last_peered":"2023-10-30T12:32:45.149747+0000","last_clean":"2023-10-30T12:32:45.149747+0000","last_became_active":"2023-10-30T12:12:02.269307+0000","last_became_peered":"2023-10-30T12:12:02.269307+0000","last_unstale":"2023-10-30T12:32:45.149747+0000","last_undegraded":"2023-10-30T12:32:45.149747+0000","last_fullsized":"2023-10-30T12:32:45.149747+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:45.148252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:45.148252+0000","last_clean_scrub_stamp":"2023-10-30T12:32:45.148252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:36:41.473826+0000","scrub_duration":0.050301950999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":864,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:40.067739+0000","last_change":"2023-10-30T12:32:40.067739+0000","last_active":"2023-10-30T12:32:40.067739+0000","last_peered":"2023-10-30T12:32:40.067739+0000","last_clean":"2023-10-30T12:32:40.067739+0000","last_became_active":"2023-10-30T12:11:55.581296+0000","last_became_peered":"2023-10-30T12:11:55.581296+0000","last_unstale":"2023-10-30T12:32:40.067739+0000","last_undegraded":"2023-10-30T12:32:40.067739+0000","last_fullsized":"2023-10-30T12:32:40.067739+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:40.063585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:40.063585+0000","last_clean_scrub_stamp":"2023-10-30T12:32:40.063585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:43.615070+0000","scrub_duration":0.072154391999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:53.883296+0000","last_change":"2023-10-30T12:32:53.888028+0000","last_active":"2023-10-30T12:32:53.883296+0000","last_peered":"2023-10-30T12:32:53.883296+0000","last_clean":"2023-10-30T12:32:53.883296+0000","last_became_active":"2023-10-30T12:12:25.833774+0000","last_became_peered":"2023-10-30T12:12:25.833774+0000","last_unstale":"2023-10-30T12:32:53.883296+0000","last_undegraded":"2023-10-30T12:32:53.883296+0000","last_fullsized":"2023-10-30T12:32:53.883296+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:53.881823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:53.881823+0000","last_clean_scrub_stamp":"2023-10-30T12:32:53.881823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:47:51.656882+0000","scrub_duration":0.054464448999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:52.866011+0000","last_change":"2023-10-30T12:32:52.867084+0000","last_active":"2023-10-30T12:32:52.866011+0000","last_peered":"2023-10-30T12:32:52.866011+0000","last_clean":"2023-10-30T12:32:52.866011+0000","last_became_active":"2023-10-30T12:12:09.094001+0000","last_became_peered":"2023-10-30T12:12:09.094001+0000","last_unstale":"2023-10-30T12:32:52.866011+0000","last_undegraded":"2023-10-30T12:32:52.866011+0000","last_fullsized":"2023-10-30T12:32:52.866011+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:52.864567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:52.864567+0000","last_clean_scrub_stamp":"2023-10-30T12:32:52.864567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:41:01.102396+0000","scrub_duration":0.062358990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":878,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:50.125424+0000","last_change":"2023-10-30T12:32:50.129322+0000","last_active":"2023-10-30T12:32:50.125424+0000","last_peered":"2023-10-30T12:32:50.125424+0000","last_clean":"2023-10-30T12:32:50.125424+0000","last_became_active":"2023-10-30T12:12:04.077824+0000","last_became_peered":"2023-10-30T12:12:04.077824+0000","last_unstale":"2023-10-30T12:32:50.125424+0000","last_undegraded":"2023-10-30T12:32:50.125424+0000","last_fullsized":"2023-10-30T12:32:50.125424+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:50.124037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:50.124037+0000","last_clean_scrub_stamp":"2023-10-30T12:32:50.124037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:43:02.338696+0000","scrub_duration":0.101601231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":852,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.538113+0000","last_change":"2023-10-30T12:11:50.993004+0000","last_active":"2023-10-30T12:32:33.538113+0000","last_peered":"2023-10-30T12:32:33.538113+0000","last_clean":"2023-10-30T12:32:33.538113+0000","last_became_active":"2023-10-30T12:11:50.987299+0000","last_became_peered":"2023-10-30T12:11:50.987299+0000","last_unstale":"2023-10-30T12:32:33.538113+0000","last_undegraded":"2023-10-30T12:32:33.538113+0000","last_fullsized":"2023-10-30T12:32:33.538113+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":853,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.526899+0000","last_change":"2023-10-30T12:11:51.028381+0000","last_active":"2023-10-30T12:32:33.526899+0000","last_peered":"2023-10-30T12:32:33.526899+0000","last_clean":"2023-10-30T12:32:33.526899+0000","last_became_active":"2023-10-30T12:11:51.023075+0000","last_became_peered":"2023-10-30T12:11:51.023075+0000","last_unstale":"2023-10-30T12:32:33.526899+0000","last_undegraded":"2023-10-30T12:32:33.526899+0000","last_fullsized":"2023-10-30T12:32:33.526899+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":884,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:41.613858+0000","last_change":"2023-10-30T12:32:41.614961+0000","last_active":"2023-10-30T12:32:41.613858+0000","last_peered":"2023-10-30T12:32:41.613858+0000","last_clean":"2023-10-30T12:32:41.613858+0000","last_became_active":"2023-10-30T12:10:17.738241+0000","last_became_peered":"2023-10-30T12:10:17.738241+0000","last_unstale":"2023-10-30T12:32:41.613858+0000","last_undegraded":"2023-10-30T12:32:41.613858+0000","last_fullsized":"2023-10-30T12:32:41.613858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:41.612385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:41.612385+0000","last_clean_scrub_stamp":"2023-10-30T12:32:41.612385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:18:13.401922+0000","scrub_duration":0.035484615999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":870,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:39.895283+0000","last_change":"2023-10-30T12:32:39.896413+0000","last_active":"2023-10-30T12:32:39.895283+0000","last_peered":"2023-10-30T12:32:39.895283+0000","last_clean":"2023-10-30T12:32:39.895283+0000","last_became_active":"2023-10-30T12:11:52.257573+0000","last_became_peered":"2023-10-30T12:11:52.257573+0000","last_unstale":"2023-10-30T12:32:39.895283+0000","last_undegraded":"2023-10-30T12:32:39.895283+0000","last_fullsized":"2023-10-30T12:32:39.895283+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:39.893840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:39.893840+0000","last_clean_scrub_stamp":"2023-10-30T12:32:39.893840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:04.201929+0000","scrub_duration":0.031836309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":885,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:40.115821+0000","last_change":"2023-10-30T12:32:40.116902+0000","last_active":"2023-10-30T12:32:40.115821+0000","last_peered":"2023-10-30T12:32:40.115821+0000","last_clean":"2023-10-30T12:32:40.115821+0000","last_became_active":"2023-10-30T12:10:17.763227+0000","last_became_peered":"2023-10-30T12:10:17.763227+0000","last_unstale":"2023-10-30T12:32:40.115821+0000","last_undegraded":"2023-10-30T12:32:40.115821+0000","last_fullsized":"2023-10-30T12:32:40.115821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:40.114411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:40.114411+0000","last_clean_scrub_stamp":"2023-10-30T12:32:40.114411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:05.152002+0000","scrub_duration":0.037475106000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":894,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:50.536434+0000","last_change":"2023-10-30T12:32:50.537406+0000","last_active":"2023-10-30T12:32:50.536434+0000","last_peered":"2023-10-30T12:32:50.536434+0000","last_clean":"2023-10-30T12:32:50.536434+0000","last_became_active":"2023-10-30T12:11:55.613173+0000","last_became_peered":"2023-10-30T12:11:55.613173+0000","last_unstale":"2023-10-30T12:32:50.536434+0000","last_undegraded":"2023-10-30T12:32:50.536434+0000","last_fullsized":"2023-10-30T12:32:50.536434+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:50.535095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:50.535095+0000","last_clean_scrub_stamp":"2023-10-30T12:32:50.535095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:14.593021+0000","scrub_duration":0.061830881999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":884,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:44.659407+0000","last_change":"2023-10-30T12:32:44.660493+0000","last_active":"2023-10-30T12:32:44.659407+0000","last_peered":"2023-10-30T12:32:44.659407+0000","last_clean":"2023-10-30T12:32:44.659407+0000","last_became_active":"2023-10-30T12:10:16.907205+0000","last_became_peered":"2023-10-30T12:10:16.907205+0000","last_unstale":"2023-10-30T12:32:44.659407+0000","last_undegraded":"2023-10-30T12:32:44.659407+0000","last_fullsized":"2023-10-30T12:32:44.659407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:44.658017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:44.658017+0000","last_clean_scrub_stamp":"2023-10-30T12:32:44.658017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:04.891705+0000","scrub_duration":0.033505479999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":871,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:37.904809+0000","last_change":"2023-10-30T12:32:37.905916+0000","last_active":"2023-10-30T12:32:37.904809+0000","last_peered":"2023-10-30T12:32:37.904809+0000","last_clean":"2023-10-30T12:32:37.904809+0000","last_became_active":"2023-10-30T12:11:52.909795+0000","last_became_peered":"2023-10-30T12:11:52.909795+0000","last_unstale":"2023-10-30T12:32:37.904809+0000","last_undegraded":"2023-10-30T12:32:37.904809+0000","last_fullsized":"2023-10-30T12:32:37.904809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:37.903361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:37.903361+0000","last_clean_scrub_stamp":"2023-10-30T12:32:37.903361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:22.724580+0000","scrub_duration":0.049141183999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":900,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:38.541118+0000","last_change":"2023-10-30T12:32:38.545483+0000","last_active":"2023-10-30T12:32:38.541118+0000","last_peered":"2023-10-30T12:32:38.541118+0000","last_clean":"2023-10-30T12:32:38.541118+0000","last_became_active":"2023-10-30T12:10:17.582475+0000","last_became_peered":"2023-10-30T12:10:17.582475+0000","last_unstale":"2023-10-30T12:32:38.541118+0000","last_undegraded":"2023-10-30T12:32:38.541118+0000","last_fullsized":"2023-10-30T12:32:38.541118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:38.539673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:38.539673+0000","last_clean_scrub_stamp":"2023-10-30T12:32:38.539673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:59:45.700659+0000","scrub_duration":0.055551556000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":869,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:39.027151+0000","last_change":"2023-10-30T12:32:39.029970+0000","last_active":"2023-10-30T12:32:39.027151+0000","last_peered":"2023-10-30T12:32:39.027151+0000","last_clean":"2023-10-30T12:32:39.027151+0000","last_became_active":"2023-10-30T12:11:51.581905+0000","last_became_peered":"2023-10-30T12:11:51.581905+0000","last_unstale":"2023-10-30T12:32:39.027151+0000","last_undegraded":"2023-10-30T12:32:39.027151+0000","last_fullsized":"2023-10-30T12:32:39.027151+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:39.022101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:39.022101+0000","last_clean_scrub_stamp":"2023-10-30T12:32:39.022101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:10:34.443820+0000","scrub_duration":0.045951260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":884,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:42.575212+0000","last_change":"2023-10-30T12:32:42.576311+0000","last_active":"2023-10-30T12:32:42.575212+0000","last_peered":"2023-10-30T12:32:42.575212+0000","last_clean":"2023-10-30T12:32:42.575212+0000","last_became_active":"2023-10-30T12:10:17.553523+0000","last_became_peered":"2023-10-30T12:10:17.553523+0000","last_unstale":"2023-10-30T12:32:42.575212+0000","last_undegraded":"2023-10-30T12:32:42.575212+0000","last_fullsized":"2023-10-30T12:32:42.575212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:42.573746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:42.573746+0000","last_clean_scrub_stamp":"2023-10-30T12:32:42.573746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:38.805371+0000","scrub_duration":0.038013714999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":853,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.522244+0000","last_change":"2023-10-30T12:11:50.971305+0000","last_active":"2023-10-30T12:32:33.522244+0000","last_peered":"2023-10-30T12:32:33.522244+0000","last_clean":"2023-10-30T12:32:33.522244+0000","last_became_active":"2023-10-30T12:11:50.965758+0000","last_became_peered":"2023-10-30T12:11:50.965758+0000","last_unstale":"2023-10-30T12:32:33.522244+0000","last_undegraded":"2023-10-30T12:32:33.522244+0000","last_fullsized":"2023-10-30T12:32:33.522244+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":869,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:43.938190+0000","last_change":"2023-10-30T12:32:43.939195+0000","last_active":"2023-10-30T12:32:43.938190+0000","last_peered":"2023-10-30T12:32:43.938190+0000","last_clean":"2023-10-30T12:32:43.938190+0000","last_became_active":"2023-10-30T12:10:16.811711+0000","last_became_peered":"2023-10-30T12:10:16.811711+0000","last_unstale":"2023-10-30T12:32:43.938190+0000","last_undegraded":"2023-10-30T12:32:43.938190+0000","last_fullsized":"2023-10-30T12:32:43.938190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:43.935642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:43.935642+0000","last_clean_scrub_stamp":"2023-10-30T12:32:43.935642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:19:24.462349+0000","scrub_duration":0.034953691000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":861,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:56.861599+0000","last_change":"2023-10-30T12:32:56.862623+0000","last_active":"2023-10-30T12:32:56.861599+0000","last_peered":"2023-10-30T12:32:56.861599+0000","last_clean":"2023-10-30T12:32:56.861599+0000","last_became_active":"2023-10-30T12:12:02.408646+0000","last_became_peered":"2023-10-30T12:12:02.408646+0000","last_unstale":"2023-10-30T12:32:56.861599+0000","last_undegraded":"2023-10-30T12:32:56.861599+0000","last_fullsized":"2023-10-30T12:32:56.861599+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:56.860189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:56.860189+0000","last_clean_scrub_stamp":"2023-10-30T12:32:56.860189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:19.653177+0000","scrub_duration":0.040718254000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":885,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:36.670764+0000","last_change":"2023-10-30T12:32:36.671858+0000","last_active":"2023-10-30T12:32:36.670764+0000","last_peered":"2023-10-30T12:32:36.670764+0000","last_clean":"2023-10-30T12:32:36.670764+0000","last_became_active":"2023-10-30T12:10:16.849940+0000","last_became_peered":"2023-10-30T12:10:16.849940+0000","last_unstale":"2023-10-30T12:32:36.670764+0000","last_undegraded":"2023-10-30T12:32:36.670764+0000","last_fullsized":"2023-10-30T12:32:36.670764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:36.669332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:36.669332+0000","last_clean_scrub_stamp":"2023-10-30T12:32:36.669332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:53:28.553745+0000","scrub_duration":0.090791832000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":870,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:36.882845+0000","last_change":"2023-10-30T12:32:36.883963+0000","last_active":"2023-10-30T12:32:36.882845+0000","last_peered":"2023-10-30T12:32:36.882845+0000","last_clean":"2023-10-30T12:32:36.882845+0000","last_became_active":"2023-10-30T12:11:53.066917+0000","last_became_peered":"2023-10-30T12:11:53.066917+0000","last_unstale":"2023-10-30T12:32:36.882845+0000","last_undegraded":"2023-10-30T12:32:36.882845+0000","last_fullsized":"2023-10-30T12:32:36.882845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:36.881387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:36.881387+0000","last_clean_scrub_stamp":"2023-10-30T12:32:36.881387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:33:50.617590+0000","scrub_duration":0.073083355000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":884,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:38.638352+0000","last_change":"2023-10-30T12:32:38.640066+0000","last_active":"2023-10-30T12:32:38.638352+0000","last_peered":"2023-10-30T12:32:38.638352+0000","last_clean":"2023-10-30T12:32:38.638352+0000","last_became_active":"2023-10-30T12:10:17.153120+0000","last_became_peered":"2023-10-30T12:10:17.153120+0000","last_unstale":"2023-10-30T12:32:38.638352+0000","last_undegraded":"2023-10-30T12:32:38.638352+0000","last_fullsized":"2023-10-30T12:32:38.638352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:38.636075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:38.636075+0000","last_clean_scrub_stamp":"2023-10-30T12:32:38.636075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:06:44.125261+0000","scrub_duration":0.047870970999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"900'78","reported_seq":979,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:39.080995+0000","last_change":"2023-10-30T12:32:39.085226+0000","last_active":"2023-10-30T12:32:39.080995+0000","last_peered":"2023-10-30T12:32:39.080995+0000","last_clean":"2023-10-30T12:32:39.080995+0000","last_became_active":"2023-10-30T12:10:09.748373+0000","last_became_peered":"2023-10-30T12:10:09.748373+0000","last_unstale":"2023-10-30T12:32:39.080995+0000","last_undegraded":"2023-10-30T12:32:39.080995+0000","last_fullsized":"2023-10-30T12:32:39.080995+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"900'78","last_scrub_stamp":"2023-10-30T12:32:39.079215+0000","last_deep_scrub":"900'78","last_deep_scrub_stamp":"2023-10-30T12:32:39.079215+0000","last_clean_scrub_stamp":"2023-10-30T12:32:39.079215+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:31.600901+0000","scrub_duration":0.127691637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":871,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:50.920526+0000","last_change":"2023-10-30T12:32:50.920526+0000","last_active":"2023-10-30T12:32:50.920526+0000","last_peered":"2023-10-30T12:32:50.920526+0000","last_clean":"2023-10-30T12:32:50.920526+0000","last_became_active":"2023-10-30T12:12:03.631410+0000","last_became_peered":"2023-10-30T12:12:03.631410+0000","last_unstale":"2023-10-30T12:32:50.920526+0000","last_undegraded":"2023-10-30T12:32:50.920526+0000","last_fullsized":"2023-10-30T12:32:50.920526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:50.918001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:50.918001+0000","last_clean_scrub_stamp":"2023-10-30T12:32:50.918001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:09:50.307402+0000","scrub_duration":0.056243267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":880,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:40.819762+0000","last_change":"2023-10-30T12:32:40.819762+0000","last_active":"2023-10-30T12:32:40.819762+0000","last_peered":"2023-10-30T12:32:40.819762+0000","last_clean":"2023-10-30T12:32:40.819762+0000","last_became_active":"2023-10-30T12:12:01.926913+0000","last_became_peered":"2023-10-30T12:12:01.926913+0000","last_unstale":"2023-10-30T12:32:40.819762+0000","last_undegraded":"2023-10-30T12:32:40.819762+0000","last_fullsized":"2023-10-30T12:32:40.819762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:40.814697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:40.814697+0000","last_clean_scrub_stamp":"2023-10-30T12:32:40.814697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:57.003617+0000","scrub_duration":0.028773494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":862,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:46.972772+0000","last_change":"2023-10-30T12:32:46.972772+0000","last_active":"2023-10-30T12:32:46.972772+0000","last_peered":"2023-10-30T12:32:46.972772+0000","last_clean":"2023-10-30T12:32:46.972772+0000","last_became_active":"2023-10-30T12:12:02.108733+0000","last_became_peered":"2023-10-30T12:12:02.108733+0000","last_unstale":"2023-10-30T12:32:46.972772+0000","last_undegraded":"2023-10-30T12:32:46.972772+0000","last_fullsized":"2023-10-30T12:32:46.972772+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:46.970477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:46.970477+0000","last_clean_scrub_stamp":"2023-10-30T12:32:46.970477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:51:24.619904+0000","scrub_duration":0.038933902999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":870,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:36.941395+0000","last_change":"2023-10-30T12:32:36.941395+0000","last_active":"2023-10-30T12:32:36.941395+0000","last_peered":"2023-10-30T12:32:36.941395+0000","last_clean":"2023-10-30T12:32:36.941395+0000","last_became_active":"2023-10-30T12:11:52.304910+0000","last_became_peered":"2023-10-30T12:11:52.304910+0000","last_unstale":"2023-10-30T12:32:36.941395+0000","last_undegraded":"2023-10-30T12:32:36.941395+0000","last_fullsized":"2023-10-30T12:32:36.941395+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:36.936947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:36.936947+0000","last_clean_scrub_stamp":"2023-10-30T12:32:36.936947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:37:08.108230+0000","scrub_duration":0.035594848999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":868,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.043680+0000","last_change":"2023-10-30T12:12:10.797605+0000","last_active":"2023-10-30T12:32:33.043680+0000","last_peered":"2023-10-30T12:32:33.043680+0000","last_clean":"2023-10-30T12:32:33.043680+0000","last_became_active":"2023-10-30T12:12:10.791950+0000","last_became_peered":"2023-10-30T12:12:10.791950+0000","last_unstale":"2023-10-30T12:32:33.043680+0000","last_undegraded":"2023-10-30T12:32:33.043680+0000","last_fullsized":"2023-10-30T12:32:33.043680+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":879,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:41.850865+0000","last_change":"2023-10-30T12:32:41.851869+0000","last_active":"2023-10-30T12:32:41.850865+0000","last_peered":"2023-10-30T12:32:41.850865+0000","last_clean":"2023-10-30T12:32:41.850865+0000","last_became_active":"2023-10-30T12:11:56.049790+0000","last_became_peered":"2023-10-30T12:11:56.049790+0000","last_unstale":"2023-10-30T12:32:41.850865+0000","last_undegraded":"2023-10-30T12:32:41.850865+0000","last_fullsized":"2023-10-30T12:32:41.850865+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:41.849413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:41.849413+0000","last_clean_scrub_stamp":"2023-10-30T12:32:41.849413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:56:56.410162+0000","scrub_duration":0.037757672999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":853,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.525363+0000","last_change":"2023-10-30T12:12:02.443381+0000","last_active":"2023-10-30T12:32:33.525363+0000","last_peered":"2023-10-30T12:32:33.525363+0000","last_clean":"2023-10-30T12:32:33.525363+0000","last_became_active":"2023-10-30T12:12:02.435431+0000","last_became_peered":"2023-10-30T12:12:02.435431+0000","last_unstale":"2023-10-30T12:32:33.525363+0000","last_undegraded":"2023-10-30T12:32:33.525363+0000","last_fullsized":"2023-10-30T12:32:33.525363+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":854,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:39.617738+0000","last_change":"2023-10-30T12:11:51.298662+0000","last_active":"2023-10-30T12:32:39.617738+0000","last_peered":"2023-10-30T12:32:39.617738+0000","last_clean":"2023-10-30T12:32:39.617738+0000","last_became_active":"2023-10-30T12:11:51.289446+0000","last_became_peered":"2023-10-30T12:11:51.289446+0000","last_unstale":"2023-10-30T12:32:39.617738+0000","last_undegraded":"2023-10-30T12:32:39.617738+0000","last_fullsized":"2023-10-30T12:32:39.617738+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":854,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:33.523724+0000","last_change":"2023-10-30T12:11:53.122439+0000","last_active":"2023-10-30T12:32:33.523724+0000","last_peered":"2023-10-30T12:32:33.523724+0000","last_clean":"2023-10-30T12:32:33.523724+0000","last_became_active":"2023-10-30T12:11:53.110390+0000","last_became_peered":"2023-10-30T12:11:53.110390+0000","last_unstale":"2023-10-30T12:32:33.523724+0000","last_undegraded":"2023-10-30T12:32:33.523724+0000","last_fullsized":"2023-10-30T12:32:33.523724+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:11:43.281097+0000","last_clean_scrub_stamp":"2023-10-30T12:11:43.281097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":878,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:44.117738+0000","last_change":"2023-10-30T12:32:44.118805+0000","last_active":"2023-10-30T12:32:44.117738+0000","last_peered":"2023-10-30T12:32:44.117738+0000","last_clean":"2023-10-30T12:32:44.117738+0000","last_became_active":"2023-10-30T12:12:00.784060+0000","last_became_peered":"2023-10-30T12:12:00.784060+0000","last_unstale":"2023-10-30T12:32:44.117738+0000","last_undegraded":"2023-10-30T12:32:44.117738+0000","last_fullsized":"2023-10-30T12:32:44.117738+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:44.116345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:44.116345+0000","last_clean_scrub_stamp":"2023-10-30T12:32:44.116345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:01:13.298323+0000","scrub_duration":0.041930035999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":878,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:45.123387+0000","last_change":"2023-10-30T12:32:45.124352+0000","last_active":"2023-10-30T12:32:45.123387+0000","last_peered":"2023-10-30T12:32:45.123387+0000","last_clean":"2023-10-30T12:32:45.123387+0000","last_became_active":"2023-10-30T12:12:01.679359+0000","last_became_peered":"2023-10-30T12:12:01.679359+0000","last_unstale":"2023-10-30T12:32:45.123387+0000","last_undegraded":"2023-10-30T12:32:45.123387+0000","last_fullsized":"2023-10-30T12:32:45.123387+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:45.121746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:45.121746+0000","last_clean_scrub_stamp":"2023-10-30T12:32:45.121746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:47:20.330160+0000","scrub_duration":0.048729416999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:50.798358+0000","last_change":"2023-10-30T12:32:50.799469+0000","last_active":"2023-10-30T12:32:50.798358+0000","last_peered":"2023-10-30T12:32:50.798358+0000","last_clean":"2023-10-30T12:32:50.798358+0000","last_became_active":"2023-10-30T12:12:08.259922+0000","last_became_peered":"2023-10-30T12:12:08.259922+0000","last_unstale":"2023-10-30T12:32:50.798358+0000","last_undegraded":"2023-10-30T12:32:50.798358+0000","last_fullsized":"2023-10-30T12:32:50.798358+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:50.794996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:50.794996+0000","last_clean_scrub_stamp":"2023-10-30T12:32:50.794996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:00.183382+0000","scrub_duration":0.04023127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477013,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78044,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293796,"statfs":{"total":96636764160,"available":96556847104,"internally_reserved":0,"allocated":6770688,"data_stored":3198410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73136584},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:32:57 2023","interfaces":[{"interface":"back","average":{"1min":10.19,"5min":15.43,"15min":18.722999999999999},"min":{"1min":5.9960000000000004,"5min":5.2220000000000004,"15min":4.1319999999999997},"max":{"1min":27.266999999999999,"5min":120.447,"15min":213.76499999999999},"last":11.481999999999999},{"interface":"front","average":{"1min":14.603999999999999,"5min":17.568000000000001,"15min":22.356000000000002},"min":{"1min":5.1699999999999999,"5min":4.6660000000000004,"15min":4.1360000000000001},"max":{"1min":39.094999999999999,"5min":130.48699999999999,"15min":255.06999999999999},"last":9.6170000000000009}]},{"osd":1,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":16.192,"5min":19.501000000000001,"15min":21.971},"min":{"1min":6.9690000000000003,"5min":5.3449999999999998,"15min":4.6609999999999996},"max":{"1min":60.024999999999999,"5min":119.535,"15min":164.755},"last":8.3300000000000001},{"interface":"front","average":{"1min":16.702999999999999,"5min":19.878,"15min":25.541},"min":{"1min":7.5990000000000002,"5min":6.6399999999999997,"15min":6.0990000000000002},"max":{"1min":72.597999999999999,"5min":146.874,"15min":260.19099999999997},"last":12.368}]},{"osd":2,"last update":"Mon Oct 30 12:32:57 2023","interfaces":[{"interface":"back","average":{"1min":15.801,"5min":22.035,"15min":30.111000000000001},"min":{"1min":8.4570000000000007,"5min":7.1929999999999996,"15min":6.4589999999999996},"max":{"1min":40.378,"5min":215.48400000000001,"15min":234.21299999999999},"last":13.068},{"interface":"front","average":{"1min":15.904,"5min":22.300000000000001,"15min":29.846},"min":{"1min":8.4260000000000002,"5min":6.5860000000000003,"15min":6.5860000000000003},"max":{"1min":45.652000000000001,"5min":254.81999999999999,"15min":254.81999999999999},"last":15.343999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:32:57 2023","interfaces":[{"interface":"back","average":{"1min":23.488,"5min":20.946000000000002,"15min":28.969999999999999},"min":{"1min":9.0549999999999997,"5min":8.7599999999999998,"15min":7.8780000000000001},"max":{"1min":203.12100000000001,"5min":203.12100000000001,"15min":287.06200000000001},"last":17.079000000000001},{"interface":"front","average":{"1min":26.443000000000001,"5min":23.152999999999999,"15min":32.545999999999999},"min":{"1min":13.247999999999999,"5min":11.537000000000001,"15min":10.587},"max":{"1min":188.25299999999999,"5min":188.25299999999999,"15min":259.279},"last":19.007999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":20.829000000000001,"5min":28.876999999999999,"15min":35.593000000000004},"min":{"1min":12.195,"5min":11.795,"15min":8.2550000000000008},"max":{"1min":74.197999999999993,"5min":324.26600000000002,"15min":324.26600000000002},"last":17.984999999999999},{"interface":"front","average":{"1min":15.435,"5min":19.82,"15min":24.088000000000001},"min":{"1min":7.1639999999999997,"5min":6.6900000000000004,"15min":6.5679999999999996},"max":{"1min":70.387,"5min":251.57900000000001,"15min":251.57900000000001},"last":10.32}]},{"osd":6,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":21.481999999999999,"5min":28.734000000000002,"15min":28.196999999999999},"min":{"1min":12.811999999999999,"5min":10.584,"15min":7.9370000000000003},"max":{"1min":68.438999999999993,"5min":135.005,"15min":218.80600000000001},"last":20.966999999999999},{"interface":"front","average":{"1min":13.765000000000001,"5min":20.492000000000001,"15min":27.355},"min":{"1min":9.7479999999999993,"5min":8.0120000000000005,"15min":6.8170000000000002},"max":{"1min":25.957999999999998,"5min":133.999,"15min":239.80500000000001},"last":13.898}]},{"osd":7,"last update":"Mon Oct 30 12:31:57 2023","interfaces":[{"interface":"back","average":{"1min":22.760999999999999,"5min":33.521999999999998,"15min":40.765000000000001},"min":{"1min":13.420999999999999,"5min":13.420999999999999,"15min":10.132999999999999},"max":{"1min":67.742999999999995,"5min":323.58199999999999,"15min":1077.606},"last":19.853999999999999},{"interface":"front","average":{"1min":15.550000000000001,"5min":26.276,"15min":29.468},"min":{"1min":9.5069999999999997,"5min":9.0050000000000008,"15min":8.5879999999999992},"max":{"1min":21.239000000000001,"5min":253.04599999999999,"15min":253.04599999999999},"last":14.565}]}]},{"osd":2,"up_from":16,"seq":68719477012,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60000,"kb_used_data":6616,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311840,"statfs":{"total":96636764160,"available":96575324160,"internally_reserved":0,"allocated":6774784,"data_stored":3198415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:32:52 2023","interfaces":[{"interface":"back","average":{"1min":13.906000000000001,"5min":16.802,"15min":20.882000000000001},"min":{"1min":6.21,"5min":4.5149999999999997,"15min":4.5149999999999997},"max":{"1min":81.200000000000003,"5min":205.785,"15min":398.01499999999999},"last":12.798},{"interface":"front","average":{"1min":10.271000000000001,"5min":17.238,"15min":20.475000000000001},"min":{"1min":5.851,"5min":5.851,"15min":4.4790000000000001},"max":{"1min":18.594000000000001,"5min":135.32499999999999,"15min":376.49299999999999},"last":8.0760000000000005}]},{"osd":1,"last update":"Mon Oct 30 12:32:52 2023","interfaces":[{"interface":"back","average":{"1min":12.113,"5min":17.210999999999999,"15min":22.742000000000001},"min":{"1min":8.2620000000000005,"5min":7.2519999999999998,"15min":6.9660000000000002},"max":{"1min":20.853999999999999,"5min":172.09999999999999,"15min":304.97500000000002},"last":13.481999999999999},{"interface":"front","average":{"1min":24.577000000000002,"5min":23.436,"15min":29.164000000000001},"min":{"1min":10.419,"5min":9.9280000000000008,"15min":7.9989999999999997},"max":{"1min":181.94900000000001,"5min":183.803,"15min":355.99400000000003},"last":19.850999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:32:52 2023","interfaces":[{"interface":"back","average":{"1min":16.068000000000001,"5min":22.236999999999998,"15min":25.489000000000001},"min":{"1min":8.7219999999999995,"5min":8.7219999999999995,"15min":6.4080000000000004},"max":{"1min":42.259,"5min":207.35300000000001,"15min":412.42899999999997},"last":14.1},{"interface":"front","average":{"1min":12.247,"5min":16.704000000000001,"15min":24.146999999999998},"min":{"1min":7.3929999999999998,"5min":6.1959999999999997,"15min":6.1959999999999997},"max":{"1min":19.873000000000001,"5min":147.63900000000001,"15min":328.90899999999999},"last":9.7620000000000005}]},{"osd":4,"last update":"Mon Oct 30 12:32:52 2023","interfaces":[{"interface":"back","average":{"1min":20.837,"5min":17.399000000000001,"15min":20.338999999999999},"min":{"1min":6.9390000000000001,"5min":6.9390000000000001,"15min":6.5439999999999996},"max":{"1min":200.41300000000001,"5min":200.41300000000001,"15min":326.565},"last":11.561},{"interface":"front","average":{"1min":19.346,"5min":19.966999999999999,"15min":24.449000000000002},"min":{"1min":10.882999999999999,"5min":10.289,"15min":9.8710000000000004},"max":{"1min":45.310000000000002,"5min":83.744,"15min":302.76499999999999},"last":12.885999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:32:52 2023","interfaces":[{"interface":"back","average":{"1min":31.780999999999999,"5min":22.683,"15min":21.606999999999999},"min":{"1min":9.7829999999999995,"5min":5.4139999999999997,"15min":4.9539999999999997},"max":{"1min":211.773,"5min":211.773,"15min":246.011},"last":18.411999999999999},{"interface":"front","average":{"1min":17.983000000000001,"5min":15.692,"15min":26.936},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":208.166,"5min":208.166,"15min":350.80399999999997},"last":7.2140000000000004}]},{"osd":6,"last update":"Mon Oct 30 12:32:52 2023","interfaces":[{"interface":"back","average":{"1min":24.526,"5min":22.323,"15min":25.875},"min":{"1min":10.798,"5min":9.5139999999999993,"15min":7.8949999999999996},"max":{"1min":170.02799999999999,"5min":170.02799999999999,"15min":374.16699999999997},"last":15.414},{"interface":"front","average":{"1min":26.651,"5min":24.942,"15min":28.331},"min":{"1min":9.391,"5min":9.391,"15min":8.2400000000000002},"max":{"1min":184.88999999999999,"5min":184.88999999999999,"15min":333.37299999999999},"last":20.521000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:32:52 2023","interfaces":[{"interface":"back","average":{"1min":12.891,"5min":14.82,"15min":20.096},"min":{"1min":7.6379999999999999,"5min":6.8049999999999997,"15min":6.1539999999999999},"max":{"1min":23.285,"5min":88.486999999999995,"15min":272.11700000000002},"last":9.0020000000000007},{"interface":"front","average":{"1min":16.129999999999999,"5min":19.026,"15min":27.291},"min":{"1min":11.071,"5min":9.7449999999999992,"15min":7.4480000000000004},"max":{"1min":32.222000000000001,"5min":114.895,"15min":453.28399999999999},"last":18.096}]}]},{"osd":0,"up_from":16,"seq":68719477012,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59996,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311844,"statfs":{"total":96636764160,"available":96575328256,"internally_reserved":0,"allocated":6770688,"data_stored":3198410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:32:49 2023","interfaces":[{"interface":"back","average":{"1min":17.963999999999999,"5min":19.321999999999999,"15min":18.98},"min":{"1min":7.4240000000000004,"5min":5.6520000000000001,"15min":4.79},"max":{"1min":96.656999999999996,"5min":152.46799999999999,"15min":199.46299999999999},"last":13.845000000000001},{"interface":"front","average":{"1min":14.843999999999999,"5min":17.559999999999999,"15min":19.123000000000001},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.5279999999999996},"max":{"1min":89.953999999999994,"5min":150.90100000000001,"15min":218.26300000000001},"last":8.327}]},{"osd":2,"last update":"Mon Oct 30 12:32:49 2023","interfaces":[{"interface":"back","average":{"1min":22.099,"5min":19.155999999999999,"15min":19.678000000000001},"min":{"1min":8.5709999999999997,"5min":5.9580000000000002,"15min":5.9580000000000002},"max":{"1min":102.59699999999999,"5min":120.355,"15min":150.55799999999999},"last":15.954000000000001},{"interface":"front","average":{"1min":20.047000000000001,"5min":17.795999999999999,"15min":18.312000000000001},"min":{"1min":7.3710000000000004,"5min":6.7439999999999998,"15min":5.8010000000000002},"max":{"1min":92.296000000000006,"5min":101.89100000000001,"15min":144.61099999999999},"last":9.7289999999999992}]},{"osd":3,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":26.065000000000001,"5min":20.931000000000001,"15min":20.044},"min":{"1min":6.1289999999999996,"5min":5.484,"15min":5.484},"max":{"1min":221.83099999999999,"5min":221.83099999999999,"15min":221.83099999999999},"last":9.0099999999999998},{"interface":"front","average":{"1min":31.369,"5min":21.067,"15min":21.018999999999998},"min":{"1min":9.048,"5min":6.2770000000000001,"15min":5.8129999999999997},"max":{"1min":200.738,"5min":220.589,"15min":220.589},"last":10.334}]},{"osd":4,"last update":"Mon Oct 30 12:32:49 2023","interfaces":[{"interface":"back","average":{"1min":22.135999999999999,"5min":21.305,"15min":21.172999999999998},"min":{"1min":8.75,"5min":8.75,"15min":6.5949999999999998},"max":{"1min":94.218999999999994,"5min":116.10599999999999,"15min":247.57300000000001},"last":11.012},{"interface":"front","average":{"1min":17.291,"5min":17.158999999999999,"15min":20.523},"min":{"1min":8.0489999999999995,"5min":6.7370000000000001,"15min":6.7370000000000001},"max":{"1min":80.671999999999997,"5min":115.371,"15min":267.23899999999998},"last":12.417}]},{"osd":5,"last update":"Mon Oct 30 12:32:49 2023","interfaces":[{"interface":"back","average":{"1min":18.238,"5min":19.504000000000001,"15min":21.047999999999998},"min":{"1min":7.6109999999999998,"5min":7.3899999999999997,"15min":6.7080000000000002},"max":{"1min":63.841999999999999,"5min":210.66800000000001,"15min":232.041},"last":16.815999999999999},{"interface":"front","average":{"1min":23.806000000000001,"5min":23.82,"15min":24.617000000000001},"min":{"1min":12.744,"5min":9.843,"15min":9.843},"max":{"1min":92.659999999999997,"5min":211.376,"15min":256.58699999999999},"last":18.402000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:32:49 2023","interfaces":[{"interface":"back","average":{"1min":23.859000000000002,"5min":24.050000000000001,"15min":23.568000000000001},"min":{"1min":10.741,"5min":8.2750000000000004,"15min":8.2750000000000004},"max":{"1min":99.924999999999997,"5min":136.44399999999999,"15min":147.01499999999999},"last":11.725},{"interface":"front","average":{"1min":23.611999999999998,"5min":24.963000000000001,"15min":24.300000000000001},"min":{"1min":10.833,"5min":8.4619999999999997,"15min":8.4619999999999997},"max":{"1min":71.478999999999999,"5min":196.33799999999999,"15min":196.33799999999999},"last":17.742999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:32:49 2023","interfaces":[{"interface":"back","average":{"1min":19.381,"5min":17.393000000000001,"15min":19.437999999999999},"min":{"1min":9.3360000000000003,"5min":7.2110000000000003,"15min":7.2110000000000003},"max":{"1min":90.811000000000007,"5min":95.358999999999995,"15min":180.99100000000001},"last":13.093},{"interface":"front","average":{"1min":26.015000000000001,"5min":23.294,"15min":25.687000000000001},"min":{"1min":13.375999999999999,"5min":9.1110000000000007,"15min":8.827},"max":{"1min":103.71599999999999,"5min":104.599,"15min":199.85300000000001},"last":19.452999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542421,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59996,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311844,"statfs":{"total":96636764160,"available":96575328256,"internally_reserved":0,"allocated":6770688,"data_stored":3198410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":13.044,"5min":12.593,"15min":18.446000000000002},"min":{"1min":4.7569999999999997,"5min":4.4199999999999999,"15min":3.7999999999999998},"max":{"1min":119.88,"5min":175.166,"15min":300.30399999999997},"last":5.1040000000000001},{"interface":"front","average":{"1min":11.448,"5min":11.372999999999999,"15min":20.356999999999999},"min":{"1min":4.0860000000000003,"5min":4.0570000000000004,"15min":4.0570000000000004},"max":{"1min":99.126000000000005,"5min":101.81399999999999,"15min":1113.894},"last":8.077}]},{"osd":1,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":15.704000000000001,"5min":14.545999999999999,"15min":18.707000000000001},"min":{"1min":6.649,"5min":5.1840000000000002,"15min":4.5839999999999996},"max":{"1min":61.970999999999997,"5min":70.786000000000001,"15min":282.928},"last":14.23},{"interface":"front","average":{"1min":17.155999999999999,"5min":16.393999999999998,"15min":22.126999999999999},"min":{"1min":7.1319999999999997,"5min":5.524,"15min":4.7300000000000004},"max":{"1min":80.436000000000007,"5min":105.95399999999999,"15min":331.18700000000001},"last":8.952}]},{"osd":2,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":17.294,"5min":16.800999999999998,"15min":23.338999999999999},"min":{"1min":6.3819999999999997,"5min":6.3819999999999997,"15min":5.242},"max":{"1min":56.945999999999998,"5min":96.292000000000002,"15min":309.40600000000001},"last":12.742000000000001},{"interface":"front","average":{"1min":18.611000000000001,"5min":18.437999999999999,"15min":25.881},"min":{"1min":7.7229999999999999,"5min":6.6970000000000001,"15min":5.883},"max":{"1min":120.937,"5min":172.32300000000001,"15min":318.04000000000002},"last":11.941000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":17.614999999999998,"5min":19.256,"15min":26.411000000000001},"min":{"1min":6.8799999999999999,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":96.552000000000007,"5min":211.71899999999999,"15min":419.54599999999999},"last":11.272},{"interface":"front","average":{"1min":19.911999999999999,"5min":22.864000000000001,"15min":27.788},"min":{"1min":9.8499999999999996,"5min":8.0380000000000003,"15min":7.4969999999999999},"max":{"1min":60.283000000000001,"5min":280.57400000000001,"15min":353.90800000000002},"last":15.412000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":16.817,"5min":18.376999999999999,"15min":23.847999999999999},"min":{"1min":6.069,"5min":6.069,"15min":5.7039999999999997},"max":{"1min":97.218999999999994,"5min":158.77000000000001,"15min":420.41399999999999},"last":9.8000000000000007},{"interface":"front","average":{"1min":17.460000000000001,"5min":20.013000000000002,"15min":25.613},"min":{"1min":7.5880000000000001,"5min":7.1020000000000003,"15min":6.6799999999999997},"max":{"1min":88.391999999999996,"5min":214.81,"15min":329.49000000000001},"last":13.419}]},{"osd":5,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":16.939,"5min":18.957999999999998,"15min":24.402000000000001},"min":{"1min":9.1319999999999997,"5min":8.4689999999999994,"15min":7.6449999999999996},"max":{"1min":50.881,"5min":172.79300000000001,"15min":291.26999999999998},"last":14.885},{"interface":"front","average":{"1min":20.372,"5min":22.681000000000001,"15min":29.207000000000001},"min":{"1min":13.125,"5min":11.194000000000001,"15min":11.194000000000001},"max":{"1min":51.552,"5min":173.39699999999999,"15min":310.00700000000001},"last":16.994}]},{"osd":6,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":17.547999999999998,"5min":19.128,"15min":24.890000000000001},"min":{"1min":7.8220000000000001,"5min":7.8220000000000001,"15min":6.7549999999999999},"max":{"1min":98.515000000000001,"5min":160.167,"15min":421.15699999999998},"last":10.722},{"interface":"front","average":{"1min":20.236000000000001,"5min":22.681999999999999,"15min":26.983000000000001},"min":{"1min":12.522,"5min":9.8710000000000004,"15min":9.8710000000000004},"max":{"1min":53.405999999999999,"5min":143.62899999999999,"15min":315.82299999999998},"last":17.603000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542419,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100228,"kb_used_data":6652,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94271612,"statfs":{"total":96636764160,"available":96534130688,"internally_reserved":0,"allocated":6811648,"data_stored":3596255,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":95812038},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":14.826000000000001,"5min":15.159000000000001,"15min":21.579999999999998},"min":{"1min":5.1609999999999996,"5min":5.1609999999999996,"15min":4.1980000000000004},"max":{"1min":64.741,"5min":92.221999999999994,"15min":431.12},"last":10.545},{"interface":"front","average":{"1min":17.555,"5min":15.711,"15min":21.766999999999999},"min":{"1min":6.2409999999999997,"5min":5.1420000000000003,"15min":4.7199999999999998},"max":{"1min":98.355999999999995,"5min":98.355999999999995,"15min":399.90300000000002},"last":5.7370000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":26.707000000000001,"5min":23.045999999999999,"15min":27.221},"min":{"1min":6.7359999999999998,"5min":6.7359999999999998,"15min":6.3339999999999996},"max":{"1min":162.07900000000001,"5min":162.07900000000001,"15min":284.32900000000001},"last":8.718},{"interface":"front","average":{"1min":21.553000000000001,"5min":19.859000000000002,"15min":25.832000000000001},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":5.8230000000000004},"max":{"1min":118.956,"5min":118.956,"15min":331.928},"last":8.0150000000000006}]},{"osd":3,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":23.033999999999999,"5min":24.411000000000001,"15min":31.416},"min":{"1min":8.1790000000000003,"5min":7.1970000000000001,"15min":6.0229999999999997},"max":{"1min":104.30500000000001,"5min":153.827,"15min":270.78100000000001},"last":15.093},{"interface":"front","average":{"1min":21.942,"5min":24.722000000000001,"15min":29.132000000000001},"min":{"1min":8.5429999999999993,"5min":6.633,"15min":6.4960000000000004},"max":{"1min":106.015,"5min":235.411,"15min":321.42700000000002},"last":9.9550000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":22.050000000000001,"5min":26.515000000000001,"15min":27.753},"min":{"1min":10.422000000000001,"5min":8.4659999999999993,"15min":6.6890000000000001},"max":{"1min":143.80600000000001,"5min":196.18000000000001,"15min":278.36399999999998},"last":18.106999999999999},{"interface":"front","average":{"1min":21.295999999999999,"5min":24.879999999999999,"15min":25.135999999999999},"min":{"1min":8.8339999999999996,"5min":7.641,"15min":6.3280000000000003},"max":{"1min":148.40100000000001,"5min":219.45400000000001,"15min":284.76900000000001},"last":11.204000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:32:46 2023","interfaces":[{"interface":"back","average":{"1min":17.353000000000002,"5min":20.331,"15min":23.489000000000001},"min":{"1min":8.0709999999999997,"5min":7.077,"15min":6.0869999999999997},"max":{"1min":73.831000000000003,"5min":127.604,"15min":281.173},"last":13.762},{"interface":"front","average":{"1min":26.169,"5min":26.431999999999999,"15min":26.504000000000001},"min":{"1min":10.946999999999999,"5min":10.473000000000001,"15min":8.2370000000000001},"max":{"1min":141.85499999999999,"5min":280.072,"15min":293.15800000000002},"last":11.792999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":21.413,"5min":24.916,"15min":30.504999999999999},"min":{"1min":8.9939999999999998,"5min":6.7089999999999996,"15min":6.7089999999999996},"max":{"1min":126.279,"5min":126.279,"15min":391.726},"last":17.172000000000001},{"interface":"front","average":{"1min":25.158999999999999,"5min":24.603000000000002,"15min":27.167000000000002},"min":{"1min":8.7750000000000004,"5min":8.2289999999999992,"15min":8.0549999999999997},"max":{"1min":149.857,"5min":187.96799999999999,"15min":288.44099999999997},"last":14.380000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:31:53 2023","interfaces":[{"interface":"back","average":{"1min":25.628,"5min":25.503,"15min":39.037999999999997},"min":{"1min":12.448,"5min":8.6899999999999995,"15min":8.6899999999999995},"max":{"1min":137.25299999999999,"5min":137.25299999999999,"15min":1115.931},"last":12.375},{"interface":"front","average":{"1min":29.222999999999999,"5min":32.040999999999997,"15min":39.862000000000002},"min":{"1min":13.659000000000001,"5min":10.536,"15min":9.5079999999999991},"max":{"1min":151.57300000000001,"5min":199.08099999999999,"15min":1115.2660000000001},"last":13.063000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575123,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59996,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311844,"statfs":{"total":96636764160,"available":96575328256,"internally_reserved":0,"allocated":6770688,"data_stored":3198410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:31:58 2023","interfaces":[{"interface":"back","average":{"1min":9.2989999999999995,"5min":14.662000000000001,"15min":15.426},"min":{"1min":4.0119999999999996,"5min":3.9380000000000002,"15min":3.9380000000000002},"max":{"1min":36.322000000000003,"5min":200.376,"15min":220.25700000000001},"last":5.5599999999999996},{"interface":"front","average":{"1min":9.7750000000000004,"5min":16.427,"15min":15.846},"min":{"1min":5.4180000000000001,"5min":4.6449999999999996,"15min":4.1799999999999997},"max":{"1min":28.044,"5min":196.261,"15min":219.40000000000001},"last":12.654}]},{"osd":1,"last update":"Mon Oct 30 12:32:53 2023","interfaces":[{"interface":"back","average":{"1min":21.423999999999999,"5min":19.170000000000002,"15min":18.244},"min":{"1min":8.1899999999999995,"5min":5.1180000000000003,"15min":4.306},"max":{"1min":131.17500000000001,"5min":215.86699999999999,"15min":216.90299999999999},"last":15.488},{"interface":"front","average":{"1min":20.475999999999999,"5min":20.645,"15min":18.667000000000002},"min":{"1min":7.7119999999999997,"5min":5.2039999999999997,"15min":4.5570000000000004},"max":{"1min":143.58799999999999,"5min":225.39699999999999,"15min":225.39699999999999},"last":10.847}]},{"osd":2,"last update":"Mon Oct 30 12:32:07 2023","interfaces":[{"interface":"back","average":{"1min":12.401,"5min":18.358000000000001,"15min":19.98},"min":{"1min":6.3170000000000002,"5min":5.4729999999999999,"15min":5.4729999999999999},"max":{"1min":93.713999999999999,"5min":245.411,"15min":245.411},"last":11.92},{"interface":"front","average":{"1min":14.782999999999999,"5min":22.184000000000001,"15min":22.800000000000001},"min":{"1min":8.9339999999999993,"5min":6.7679999999999998,"15min":6.7679999999999998},"max":{"1min":26.739999999999998,"5min":244.76400000000001,"15min":244.76400000000001},"last":18.413}]},{"osd":3,"last update":"Mon Oct 30 12:31:58 2023","interfaces":[{"interface":"back","average":{"1min":14.455,"5min":25.152000000000001,"15min":22.244},"min":{"1min":7.1970000000000001,"5min":5.9649999999999999,"15min":5.9649999999999999},"max":{"1min":35.399999999999999,"5min":217.495,"15min":300.44600000000003},"last":14.679},{"interface":"front","average":{"1min":17.216999999999999,"5min":26.016999999999999,"15min":23.670000000000002},"min":{"1min":10.24,"5min":9.6560000000000006,"15min":8.4960000000000004},"max":{"1min":31.969000000000001,"5min":249.18100000000001,"15min":249.18100000000001},"last":13.340999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:32:53 2023","interfaces":[{"interface":"back","average":{"1min":21.369,"5min":20.936,"15min":22.457000000000001},"min":{"1min":7.5759999999999996,"5min":7.5759999999999996,"15min":6.556},"max":{"1min":105.456,"5min":246.57599999999999,"15min":246.57599999999999},"last":19.588000000000001},{"interface":"front","average":{"1min":14.884,"5min":18.949999999999999,"15min":19.352},"min":{"1min":6.5640000000000001,"5min":6.5640000000000001,"15min":6.5209999999999999},"max":{"1min":52.646999999999998,"5min":233.952,"15min":233.952},"last":8.5519999999999996}]},{"osd":5,"last update":"Mon Oct 30 12:32:34 2023","interfaces":[{"interface":"back","average":{"1min":29.417999999999999,"5min":24.599,"15min":23.084},"min":{"1min":10,"5min":9.3940000000000001,"15min":7.4080000000000004},"max":{"1min":149.303,"5min":267.37799999999999,"15min":267.37799999999999},"last":20.524000000000001},{"interface":"front","average":{"1min":26.631,"5min":21.341000000000001,"15min":21.408999999999999},"min":{"1min":8.3149999999999995,"5min":7.0170000000000003,"15min":5.9089999999999998},"max":{"1min":139.23400000000001,"5min":251.63499999999999,"15min":251.63499999999999},"last":10.035}]},{"osd":7,"last update":"Mon Oct 30 12:32:34 2023","interfaces":[{"interface":"back","average":{"1min":31.233000000000001,"5min":25.446000000000002,"15min":23.885999999999999},"min":{"1min":11.182,"5min":9.8510000000000009,"15min":5.7359999999999998},"max":{"1min":142.589,"5min":240.048,"15min":240.048},"last":17.443000000000001},{"interface":"front","average":{"1min":24.672000000000001,"5min":24.073,"15min":22.902999999999999},"min":{"1min":9.9990000000000006,"5min":7.7110000000000003,"15min":7.4089999999999998},"max":{"1min":166.40100000000001,"5min":250.70400000000001,"15min":250.70400000000001},"last":16.170000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575123,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78080,"kb_used_data":6648,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293760,"statfs":{"total":96636764160,"available":96556810240,"internally_reserved":0,"allocated":6807552,"data_stored":3596250,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:32:56 2023","interfaces":[{"interface":"back","average":{"1min":17.135999999999999,"5min":16.789000000000001,"15min":14.869},"min":{"1min":4.3869999999999996,"5min":4.0549999999999997,"15min":3.9620000000000002},"max":{"1min":154.18000000000001,"5min":154.18000000000001,"15min":154.18000000000001},"last":4.5789999999999997},{"interface":"front","average":{"1min":15.452,"5min":15.824,"15min":14.702},"min":{"1min":6.625,"5min":5.1280000000000001,"15min":4.3170000000000002},"max":{"1min":85.367000000000004,"5min":128,"15min":128},"last":12.646000000000001}]},{"osd":1,"last update":"Mon Oct 30 12:32:56 2023","interfaces":[{"interface":"back","average":{"1min":19.474,"5min":21.721,"15min":19.373999999999999},"min":{"1min":5.282,"5min":5.282,"15min":5.0410000000000004},"max":{"1min":93.477999999999994,"5min":213.06700000000001,"15min":235.05199999999999},"last":9.0410000000000004},{"interface":"front","average":{"1min":20.033000000000001,"5min":21.166,"15min":18.876999999999999},"min":{"1min":5.9589999999999996,"5min":5.9589999999999996,"15min":5.2210000000000001},"max":{"1min":84.725999999999999,"5min":187.249,"15min":420.82900000000001},"last":13.445}]},{"osd":2,"last update":"Mon Oct 30 12:32:56 2023","interfaces":[{"interface":"back","average":{"1min":25.391999999999999,"5min":19.021999999999998,"15min":19.715},"min":{"1min":10.388999999999999,"5min":7.4199999999999999,"15min":5.9539999999999997},"max":{"1min":194.97999999999999,"5min":194.97999999999999,"15min":194.97999999999999},"last":18.341999999999999},{"interface":"front","average":{"1min":21.866,"5min":15.571999999999999,"15min":16.978999999999999},"min":{"1min":6.2640000000000002,"5min":4.8879999999999999,"15min":4.7450000000000001},"max":{"1min":204.173,"5min":204.173,"15min":204.173},"last":9.5999999999999996}]},{"osd":3,"last update":"Mon Oct 30 12:31:59 2023","interfaces":[{"interface":"back","average":{"1min":13.568,"5min":17.634,"15min":21.957999999999998},"min":{"1min":6.3159999999999998,"5min":6.2089999999999996,"15min":5.0739999999999998},"max":{"1min":52.534999999999997,"5min":110.496,"15min":234.13999999999999},"last":11.052},{"interface":"front","average":{"1min":17.343,"5min":19.657,"15min":24.550999999999998},"min":{"1min":8.4339999999999993,"5min":6.8070000000000004,"15min":6.5330000000000004},"max":{"1min":58.048000000000002,"5min":135.05500000000001,"15min":232.31700000000001},"last":11.712}]},{"osd":4,"last update":"Mon Oct 30 12:32:56 2023","interfaces":[{"interface":"back","average":{"1min":19.800999999999998,"5min":19.067,"15min":20.035},"min":{"1min":9.1899999999999995,"5min":7.6159999999999997,"15min":6.1299999999999999},"max":{"1min":115.056,"5min":115.056,"15min":265.50400000000002},"last":15.090999999999999},{"interface":"front","average":{"1min":23.84,"5min":20.538,"15min":19.704000000000001},"min":{"1min":10.19,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":163.90100000000001,"5min":163.90100000000001,"15min":163.90100000000001},"last":15.818}]},{"osd":6,"last update":"Mon Oct 30 12:32:56 2023","interfaces":[{"interface":"back","average":{"1min":22.785,"5min":21.588000000000001,"15min":19.966999999999999},"min":{"1min":11.532999999999999,"5min":8.1229999999999993,"15min":8.1229999999999993},"max":{"1min":114.43899999999999,"5min":114.43899999999999,"15min":150.97200000000001},"last":16.460000000000001},{"interface":"front","average":{"1min":25.847000000000001,"5min":23.702999999999999,"15min":21.257000000000001},"min":{"1min":9.9510000000000005,"5min":7.9870000000000001,"15min":7.3460000000000001},"max":{"1min":161.34800000000001,"5min":182.376,"15min":182.376},"last":17.715}]},{"osd":7,"last update":"Mon Oct 30 12:32:56 2023","interfaces":[{"interface":"back","average":{"1min":19.847999999999999,"5min":22.814,"15min":24.838000000000001},"min":{"1min":7.242,"5min":7.242,"15min":6.96},"max":{"1min":123.27200000000001,"5min":124.621,"15min":854.64300000000003},"last":10.499000000000001},{"interface":"front","average":{"1min":25.837,"5min":18.731000000000002,"15min":17.510000000000002},"min":{"1min":10.747999999999999,"5min":6.2670000000000003,"15min":5.5300000000000002},"max":{"1min":207.78,"5min":207.78,"15min":264.84699999999998},"last":14.439}]}]},{"osd":4,"up_from":13,"seq":55834575123,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82204,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289636,"statfs":{"total":96636764160,"available":96552587264,"internally_reserved":0,"allocated":6770688,"data_stored":3198410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:32:57 2023","interfaces":[{"interface":"back","average":{"1min":18.170999999999999,"5min":15.954000000000001,"15min":19.619},"min":{"1min":5.6289999999999996,"5min":5.1920000000000002,"15min":4.617},"max":{"1min":118.92400000000001,"5min":139.40199999999999,"15min":372.52300000000002},"last":6.6589999999999998},{"interface":"front","average":{"1min":23.437000000000001,"5min":20.744,"15min":23.27},"min":{"1min":8.8390000000000004,"5min":6.641,"15min":6.0110000000000001},"max":{"1min":144.375,"5min":144.375,"15min":200.37299999999999},"last":12.337999999999999}]},{"osd":1,"last update":"Mon Oct 30 12:32:32 2023","interfaces":[{"interface":"back","average":{"1min":24.058,"5min":20.231000000000002,"15min":22.433},"min":{"1min":5.5039999999999996,"5min":5.5039999999999996,"15min":5.5039999999999996},"max":{"1min":126.119,"5min":167.84700000000001,"15min":231.49799999999999},"last":15.839},{"interface":"front","average":{"1min":20.042999999999999,"5min":16.888000000000002,"15min":19.818999999999999},"min":{"1min":5.79,"5min":5.2690000000000001,"15min":4.641},"max":{"1min":112.584,"5min":223.441,"15min":223.441},"last":8.4849999999999994}]},{"osd":2,"last update":"Mon Oct 30 12:32:32 2023","interfaces":[{"interface":"back","average":{"1min":15.933999999999999,"5min":23.056000000000001,"15min":26.792000000000002},"min":{"1min":6.2549999999999999,"5min":6.2549999999999999,"15min":6.2149999999999999},"max":{"1min":55.453000000000003,"5min":198.66999999999999,"15min":438.15499999999997},"last":10.597},{"interface":"front","average":{"1min":14.124000000000001,"5min":18.303000000000001,"15min":21.728000000000002},"min":{"1min":6.5380000000000003,"5min":6.2220000000000004,"15min":4.944},"max":{"1min":41.518999999999998,"5min":254.06399999999999,"15min":254.06399999999999},"last":13.785}]},{"osd":3,"last update":"Mon Oct 30 12:32:04 2023","interfaces":[{"interface":"back","average":{"1min":17.190999999999999,"5min":23.48,"15min":28.890000000000001},"min":{"1min":10.754,"5min":6.9710000000000001,"15min":6.9710000000000001},"max":{"1min":52.381999999999998,"5min":269.06700000000001,"15min":269.06700000000001},"last":11.627000000000001},{"interface":"front","average":{"1min":11.754,"5min":16.59,"15min":22.030999999999999},"min":{"1min":4.8230000000000004,"5min":4.8230000000000004,"15min":4.8230000000000004},"max":{"1min":53.374000000000002,"5min":239.715,"15min":239.715},"last":7.6740000000000004}]},{"osd":5,"last update":"Mon Oct 30 12:32:57 2023","interfaces":[{"interface":"back","average":{"1min":24.782,"5min":22.007000000000001,"15min":23.882999999999999},"min":{"1min":7.5679999999999996,"5min":7.5679999999999996,"15min":7.5679999999999996},"max":{"1min":147.47900000000001,"5min":147.47900000000001,"15min":268.34199999999998},"last":13.051},{"interface":"front","average":{"1min":24.309000000000001,"5min":18.463000000000001,"15min":20.356999999999999},"min":{"1min":8.2379999999999995,"5min":6.4000000000000004,"15min":5.452},"max":{"1min":160.215,"5min":160.215,"15min":252.899},"last":18.68}]},{"osd":6,"last update":"Mon Oct 30 12:32:57 2023","interfaces":[{"interface":"back","average":{"1min":24.315999999999999,"5min":19.625,"15min":24.132999999999999},"min":{"1min":10.522,"5min":8.8559999999999999,"15min":7.21},"max":{"1min":91.209999999999994,"5min":93.864000000000004,"15min":273.90100000000001},"last":15.146000000000001},{"interface":"front","average":{"1min":14.233000000000001,"5min":16.449000000000002,"15min":26.591999999999999},"min":{"1min":6.5090000000000003,"5min":5.8849999999999998,"15min":5.8849999999999998},"max":{"1min":46.805999999999997,"5min":80.379000000000005,"15min":292.11099999999999},"last":9.6989999999999998}]},{"osd":7,"last update":"Mon Oct 30 12:32:57 2023","interfaces":[{"interface":"back","average":{"1min":36.566000000000003,"5min":26.021000000000001,"15min":30.43},"min":{"1min":10.157,"5min":9.0250000000000004,"15min":6.46},"max":{"1min":193.06899999999999,"5min":193.06899999999999,"15min":652.50699999999995},"last":16.904},{"interface":"front","average":{"1min":34.810000000000002,"5min":24.241,"15min":27.437999999999999},"min":{"1min":14.51,"5min":10.422000000000001,"15min":8.9309999999999992},"max":{"1min":183.166,"5min":183.166,"15min":303.19999999999999},"last":17.623999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:33:00.853 INFO:tasks.ceph:pgid 24.7 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:33:00.853 INFO:tasks.ceph:pgid 24.f last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:33:00.853 INFO:tasks.ceph:pgid 24.1d last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:33:00.854 INFO:tasks.ceph:pgid 24.1b last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:33:00.854 INFO:tasks.ceph:pgid 24.17 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:33:00.855 INFO:tasks.ceph:pgid 24.15 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:33:00.855 INFO:tasks.ceph:pgid 24.14 last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:33:00.855 INFO:tasks.ceph:pgid 24.b last_scrub_stamp 2023-10-30T12:11:43.281097+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=11, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=32, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:33:00.856 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T12:33:18.129 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:18.064 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.131 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:18.066 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.146 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:18.082 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.179 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:18.087 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.182 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:18.117 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.183 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:18.119 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.197 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:18.106 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.199 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:18.134 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.215 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:18.123 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.806 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:18.714 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.806 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:18.715 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.817 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:18.725 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.841 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:18.777 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.842 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:18.778 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.856 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:18.791 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.860 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:18.796 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.861 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:18.797 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:18.870 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:18.805 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:20.856 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:33:20.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.963+0000 7fe00d98a640 10 monclient: get_monmap_and_config 2023-10-30T12:33:20.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.963+0000 7fe00d98a640 10 monclient: build_initial_monmap 2023-10-30T12:33:20.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.963+0000 7fe00d98a640 10 monclient: monmap: 2023-10-30T12:33:20.965 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:33:20.965 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:33:20.965 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:33:20.964021+0000 2023-10-30T12:33:20.966 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:33:20.964021+0000 2023-10-30T12:33:20.966 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:33:20.966 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:33:20.966 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:33:20.967 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:33:20.967 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:33:20.967 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:33:20.967 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:33:20.967 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:20.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.963+0000 7fe00d98a640 1 Processor -- start 2023-10-30T12:33:20.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.963+0000 7fe00d98a640 1 -- start start 2023-10-30T12:33:20.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.963+0000 7fe00d98a640 10 monclient: init 2023-10-30T12:33:20.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.964+0000 7fe00d98a640 10 monclient: _reopen_session rank -1 2023-10-30T12:33:20.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.964+0000 7fe00d98a640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-10-30T12:33:20.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.964+0000 7fe00d98a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe008142e40 0x7fe0080abcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:20.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.964+0000 7fe00d98a640 10 monclient(hunting): picked mon.noname-b con 0x7fe008142e40 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:33:20.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.964+0000 7fe00d98a640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe0080ac210 0x7fe00806ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:20.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.964+0000 7fe00d98a640 10 monclient(hunting): picked mon.noname-c con 0x7fe0080ac210 addr v2:172.21.15.92:3300/0 2023-10-30T12:33:20.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.964+0000 7fe00d98a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe008070480 addr v1:172.21.15.92:6789/0 2023-10-30T12:33:20.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.964+0000 7fe00d98a640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe008143ad0 con 0x7fe008070480 2023-10-30T12:33:20.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.964+0000 7fe00d98a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:33:20.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.964+0000 7fe00d98a640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe008067b90 con 0x7fe0080ac210 2023-10-30T12:33:20.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.964+0000 7fe00d98a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:33:20.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.964+0000 7fe00d98a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe008071a90 con 0x7fe008142e40 2023-10-30T12:33:20.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.964+0000 7fe00d98a640 10 monclient(hunting): _renew_subs 2023-10-30T12:33:20.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.964+0000 7fe00d98a640 10 monclient(hunting): authenticate will time out at 2792.178421s 2023-10-30T12:33:20.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.964+0000 7fe006ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe008142e40 0x7fe0080abcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:20.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.965+0000 7fe0077fe640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe008070480 0x7fe00813f650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.3:49632/0 (socket says 172.21.15.3:49632) 2023-10-30T12:33:20.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.965+0000 7fe0077fe640 1 -- 172.21.15.3:0/503371572 learned_addr learned my addr 172.21.15.3:0/503371572 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:33:20.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.965+0000 7fe006ffd640 10 monclient(hunting): get_auth_request con 0x7fe008142e40 auth_method 0 2023-10-30T12:33:20.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.965+0000 7fe006ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:33:20.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.965+0000 7fe006ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:33:20.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.965+0000 7fe006ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:33:20.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.965+0000 7fe0067fc640 1 --2- 172.21.15.3:0/503371572 >> v2:172.21.15.92:3300/0 conn(0x7fe0080ac210 0x7fe00806ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:20.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.965+0000 7fe0067fc640 10 monclient(hunting): get_auth_request con 0x7fe0080ac210 auth_method 0 2023-10-30T12:33:20.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.965+0000 7fe0067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:33:20.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.965+0000 7fe0067fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:33:20.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.965+0000 7fe0067fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:33:20.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.965+0000 7fe006ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:33:20.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.965+0000 7fe006ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:33:20.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.965+0000 7fe006ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:33:20.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.966+0000 7fe0067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:33:20.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.966+0000 7fe0067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:33:20.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.966+0000 7fe0067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:33:20.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.967+0000 7fe006ffd640 10 monclient(hunting): handle_auth_done global_id 15623 payload 274 2023-10-30T12:33:20.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.967+0000 7fe006ffd640 1 -- 172.21.15.3:0/503371572 >> v2:172.21.15.92:3300/0 conn(0x7fe0080ac210 msgr2=0x7fe00806ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:20.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.967+0000 7fe006ffd640 1 --2- 172.21.15.3:0/503371572 >> v2:172.21.15.92:3300/0 conn(0x7fe0080ac210 0x7fe00806ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:20.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.967+0000 7fe006ffd640 1 -- 172.21.15.3:0/503371572 >> v1:172.21.15.92:6789/0 conn(0x7fe008070480 legacy=0x7fe00813f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:20.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.967+0000 7fe006ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:33:20.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.967+0000 7fe006ffd640 1 monclient: found mon.noname-b 2023-10-30T12:33:20.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.967+0000 7fe006ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:33:20.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.967+0000 7fe006ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:33:20.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.967+0000 7fe006ffd640 1 -- 172.21.15.3:0/503371572 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0080a8470 con 0x7fe008142e40 2023-10-30T12:33:20.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.969+0000 7fe006ffd640 1 --2- 172.21.15.3:0/503371572 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe008142e40 0x7fe0080abcd0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fe0000097a0 tx=0x7fe00002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7003eb3be2836e7d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:20.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe005ffb640 1 -- 172.21.15.3:0/503371572 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe000031830 con 0x7fe008142e40 2023-10-30T12:33:20.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe005ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:33:20.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe005ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:33:20.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe005ffb640 10 monclient: dump: 2023-10-30T12:33:20.985 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:33:20.986 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:33:20.986 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:20.986 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:20.986 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:33:20.986 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:33:20.987 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:33:20.987 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:33:20.987 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:33:20.987 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:20.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe005ffb640 10 monclient: _finish_auth 0 2023-10-30T12:33:20.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe005ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:33:20.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe005ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:33:20.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe005ffb640 1 -- 172.21.15.3:0/503371572 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00002d040 con 0x7fe008142e40 2023-10-30T12:33:20.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe00d98a640 5 monclient: authenticate success, global_id 15623 2023-10-30T12:33:20.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe00d98a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:33:20.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe005ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:33:20.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe00d98a640 10 monclient: get_monmap_and_config success 2023-10-30T12:33:20.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe00d98a640 4 set_mon_vals no callback set 2023-10-30T12:33:20.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe005ffb640 1 -- 172.21.15.3:0/503371572 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe000031bc0 con 0x7fe008142e40 2023-10-30T12:33:20.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe005ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:33:20.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe005ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:33:20.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe005ffb640 10 monclient: dump: 2023-10-30T12:33:20.991 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:33:20.991 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:33:20.991 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:20.992 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:20.992 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:33:20.992 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:33:20.992 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:33:20.993 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:33:20.993 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:33:20.993 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:20.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe00d98a640 10 monclient: shutdown 2023-10-30T12:33:20.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe00d98a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:33:20.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe00d98a640 1 -- 172.21.15.3:0/503371572 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe008142e40 msgr2=0x7fe0080abcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:20.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe00d98a640 1 --2- 172.21.15.3:0/503371572 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe008142e40 0x7fe0080abcd0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fe0000097a0 tx=0x7fe00002b8d0 comp rx=0 tx=0).stop 2023-10-30T12:33:20.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe00d98a640 1 -- 172.21.15.3:0/503371572 shutdown_connections 2023-10-30T12:33:20.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe00d98a640 1 --2- 172.21.15.3:0/503371572 >> v2:172.21.15.92:3300/0 conn(0x7fe0080ac210 0x7fe00806ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:20.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe00d98a640 1 --2- 172.21.15.3:0/503371572 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe008142e40 0x7fe0080abcd0 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:20.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe00d98a640 1 -- 172.21.15.3:0/503371572 >> 172.21.15.3:0/503371572 conn(0x7fe0080afb80 msgr2=0x7fe0080aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:20.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe00d98a640 1 -- 172.21.15.3:0/503371572 shutdown_connections 2023-10-30T12:33:20.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.973+0000 7fe00d98a640 1 -- 172.21.15.3:0/503371572 wait complete. 2023-10-30T12:33:20.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.974+0000 7fe00d98a640 10 monclient: build_initial_monmap 2023-10-30T12:33:20.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.974+0000 7fe00d98a640 10 monclient: monmap: 2023-10-30T12:33:20.996 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:33:20.996 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:33:20.996 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:33:20.996 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:33:20.997 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:33:20.997 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:33:20.997 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:33:20.997 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:33:20.997 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:33:20.998 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:20.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.974+0000 7fe00d98a640 1 librados: starting msgr at 2023-10-30T12:33:20.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.974+0000 7fe00d98a640 1 librados: starting objecter 2023-10-30T12:33:20.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.974+0000 7fe00d98a640 1 Processor -- start 2023-10-30T12:33:20.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 1 -- start start 2023-10-30T12:33:20.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 1 librados: setting wanted keys 2023-10-30T12:33:20.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 1 librados: calling monclient init 2023-10-30T12:33:20.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 10 monclient: init 2023-10-30T12:33:20.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 10 monclient: _reopen_session rank -1 2023-10-30T12:33:21.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:33:21.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe008070480 0x7fe00813d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 10 monclient(hunting): picked mon.noname-c con 0x7fe008070480 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:33:21.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080ac210 0x7fe00813c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 10 monclient(hunting): picked mon.noname-a con 0x7fe0080ac210 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:33:21.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe008142e40 0x7fe00813f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 10 monclient(hunting): picked mon.noname-b con 0x7fe008142e40 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:33:21.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:33:21.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe008071db0 con 0x7fe0080ac210 2023-10-30T12:33:21.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:33:21.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0081381e0 con 0x7fe008142e40 2023-10-30T12:33:21.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:33:21.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00813eb50 con 0x7fe008070480 2023-10-30T12:33:21.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 10 monclient(hunting): _renew_subs 2023-10-30T12:33:21.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.975+0000 7fe00d98a640 10 monclient(hunting): authenticate will time out at 2792.189662s 2023-10-30T12:33:21.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe0067fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080ac210 0x7fe00813c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe006ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe008070480 0x7fe00813d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe0077fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe008142e40 0x7fe00813f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe0067fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080ac210 0x7fe00813c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44150/0 (socket says 172.21.15.3:44150) 2023-10-30T12:33:21.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe0067fc640 1 -- 172.21.15.3:0/2815884575 learned_addr learned my addr 172.21.15.3:0/2815884575 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:33:21.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe0067fc640 10 monclient(hunting): get_auth_request con 0x7fe0080ac210 auth_method 0 2023-10-30T12:33:21.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe0067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:33:21.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe0067fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:33:21.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe0067fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:33:21.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe006ffd640 10 monclient(hunting): get_auth_request con 0x7fe008070480 auth_method 0 2023-10-30T12:33:21.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe006ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:33:21.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe006ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:33:21.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe006ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:33:21.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe0077fe640 10 monclient(hunting): get_auth_request con 0x7fe008142e40 auth_method 0 2023-10-30T12:33:21.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe0077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:33:21.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe0077fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:33:21.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.976+0000 7fe0077fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:33:21.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.977+0000 7fe0067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:33:21.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.977+0000 7fe0067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:33:21.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.977+0000 7fe0067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:33:21.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.977+0000 7fe006ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:33:21.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.977+0000 7fe006ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:33:21.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.977+0000 7fe006ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:33:21.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.977+0000 7fe0077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:33:21.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.977+0000 7fe0077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:33:21.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.977+0000 7fe0077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:33:21.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.979+0000 7fe0077fe640 10 monclient(hunting): handle_auth_done global_id 21913 payload 824 2023-10-30T12:33:21.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.979+0000 7fe0077fe640 1 -- 172.21.15.3:0/2815884575 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe008070480 msgr2=0x7fe00813d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.979+0000 7fe0077fe640 1 --2- 172.21.15.3:0/2815884575 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe008070480 0x7fe00813d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.979+0000 7fe0077fe640 1 -- 172.21.15.3:0/2815884575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080ac210 msgr2=0x7fe00813c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.979+0000 7fe0077fe640 1 --2- 172.21.15.3:0/2815884575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080ac210 0x7fe00813c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.979+0000 7fe0077fe640 10 monclient: _finish_hunting 0 2023-10-30T12:33:21.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.979+0000 7fe0077fe640 1 monclient: found mon.noname-b 2023-10-30T12:33:21.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.979+0000 7fe0077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:33:21.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.979+0000 7fe0077fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:33:21.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.979+0000 7fe0077fe640 1 -- 172.21.15.3:0/2815884575 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00813d520 con 0x7fe008142e40 2023-10-30T12:33:21.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.981+0000 7fe0077fe640 1 --2- 172.21.15.3:0/2815884575 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe008142e40 0x7fe00813f490 secure :-1 s=READY pgs=5329 cs=0 l=1 rev1=1 crypto rx=0x7fdff800caa0 tx=0x7fdff800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:21.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.981+0000 7fdffffff640 10 client.?.objecter ms_handle_connect 0x7fe008142e40 2023-10-30T12:33:21.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.981+0000 7fdffffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:33:21.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fdffffff640 1 -- 172.21.15.3:0/2815884575 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff800ac00 con 0x7fe008142e40 2023-10-30T12:33:21.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fdffffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:33:21.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fdffffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:33:21.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fdffffff640 10 monclient: dump: 2023-10-30T12:33:21.015 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:33:21.015 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:33:21.016 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:21.016 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:21.016 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:33:21.017 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:33:21.017 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:33:21.017 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:33:21.018 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:33:21.018 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:21.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fdffffff640 10 monclient: _finish_auth 0 2023-10-30T12:33:21.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fdffffff640 10 monclient: _check_auth_tickets 2023-10-30T12:33:21.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fdffffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:33:21.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fe00d98a640 5 monclient: authenticate success, global_id 21913 2023-10-30T12:33:21.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fdffffff640 1 -- 172.21.15.3:0/2815884575 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff800e040 con 0x7fe008142e40 2023-10-30T12:33:21.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fdffffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:33:21.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fe00d98a640 10 monclient: _renew_subs 2023-10-30T12:33:21.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fdffffff640 1 -- 172.21.15.3:0/2815884575 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff8012400 con 0x7fe008142e40 2023-10-30T12:33:21.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fe00d98a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:33:21.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fe00d98a640 1 -- 172.21.15.3:0/2815884575 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00813f070 con 0x7fe008142e40 2023-10-30T12:33:21.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fdffffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:33:21.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fdffffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:33:21.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fdffffff640 10 monclient: dump: 2023-10-30T12:33:21.023 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:33:21.023 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:33:21.023 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:21.024 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:21.024 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:33:21.024 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:33:21.025 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:33:21.025 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:33:21.025 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:33:21.026 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:21.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fe0057fa640 4 set_mon_vals no callback set 2023-10-30T12:33:21.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fe00d98a640 10 client.21913.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:33:21.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fe00d98a640 10 monclient: _renew_subs 2023-10-30T12:33:21.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fe00d98a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:33:21.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fe00d98a640 1 -- 172.21.15.3:0/2815884575 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00813c8b0 con 0x7fe008142e40 2023-10-30T12:33:21.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.985+0000 7fe00d98a640 1 librados: init done 2023-10-30T12:33:21.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.986+0000 7fe00d98a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:33:21.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.986+0000 7fe00d98a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:33:21.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.986+0000 7fe00d98a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:33:21.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:20.986+0000 7fe00d98a640 1 -- 172.21.15.3:0/2815884575 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00813d6c0 con 0x7fe008142e40 2023-10-30T12:33:21.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.005+0000 7fdffffff640 1 -- 172.21.15.3:0/2815884575 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fdff8007070 con 0x7fe008142e40 2023-10-30T12:33:21.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.005+0000 7fdffffff640 1 --2- 172.21.15.3:0/2815884575 >> v2:172.21.15.92:6800/1750155735 conn(0x7fdfe806c840 0x7fdfe806ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.005+0000 7fe006ffd640 1 --2- 172.21.15.3:0/2815884575 >> v2:172.21.15.92:6800/1750155735 conn(0x7fdfe806c840 0x7fdfe806ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.005+0000 7fe006ffd640 10 monclient: get_auth_request con 0x7fdfe806c840 auth_method 0 2023-10-30T12:33:21.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.006+0000 7fe006ffd640 1 --2- 172.21.15.3:0/2815884575 >> v2:172.21.15.92:6800/1750155735 conn(0x7fdfe806c840 0x7fdfe806ed00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe000036000 tx=0x7fe00003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:21.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.006+0000 7fdffffff640 10 client.21913.objecter ms_handle_connect 0x7fdfe806c840 2023-10-30T12:33:21.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.230+0000 7fdffffff640 1 -- 172.21.15.3:0/2815884575 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7fdff8057d70 con 0x7fe008142e40 2023-10-30T12:33:21.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.230+0000 7fdffffff640 10 client.21913.objecter ms_dispatch 0x7fe008000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:33:21.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.230+0000 7fdffffff640 3 client.21913.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:33:21.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.230+0000 7fdffffff640 3 client.21913.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:33:21.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.231+0000 7fdffffff640 20 client.21913.objecter dump_active .. 0 homeless 2023-10-30T12:33:21.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.240+0000 7fdffffff640 1 -- 172.21.15.3:0/2815884575 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdff805be10 con 0x7fe008142e40 2023-10-30T12:33:21.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.240+0000 7fdffffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:33:21.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.240+0000 7fdffffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:33:21.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.420+0000 7fe00d98a640 1 -- 172.21.15.3:0/2815884575 --> v2:172.21.15.92:6800/1750155735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe0080ac630 con 0x7fdfe806c840 2023-10-30T12:33:21.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.425+0000 7fdffffff640 1 -- 172.21.15.3:0/2815884575 <== mgr.4103 v2:172.21.15.92:6800/1750155735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151881 (secure 0 0 0) 0x7fe000009400 con 0x7fdfe806c840 2023-10-30T12:33:21.428 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-30T12:33:21.429 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-10-30T12:33:21.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 10 librados: watch_flush enter 2023-10-30T12:33:21.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 10 librados: watch_flush exit 2023-10-30T12:33:21.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 20 client.21913.objecter shutdown clearing up homeless session... 2023-10-30T12:33:21.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 10 client.21913.objecter successfully canceled tick 2023-10-30T12:33:21.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 1 -- 172.21.15.3:0/2815884575 >> v2:172.21.15.92:6800/1750155735 conn(0x7fdfe806c840 msgr2=0x7fdfe806ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 1 --2- 172.21.15.3:0/2815884575 >> v2:172.21.15.92:6800/1750155735 conn(0x7fdfe806c840 0x7fdfe806ed00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe000036000 tx=0x7fe00003c040 comp rx=0 tx=0).stop 2023-10-30T12:33:21.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 10 monclient: shutdown 2023-10-30T12:33:21.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:33:21.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 1 -- 172.21.15.3:0/2815884575 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe008142e40 msgr2=0x7fe00813f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 1 --2- 172.21.15.3:0/2815884575 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe008142e40 0x7fe00813f490 secure :-1 s=READY pgs=5329 cs=0 l=1 rev1=1 crypto rx=0x7fdff800caa0 tx=0x7fdff800cf60 comp rx=0 tx=0).stop 2023-10-30T12:33:21.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 1 -- 172.21.15.3:0/2815884575 shutdown_connections 2023-10-30T12:33:21.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 1 --2- 172.21.15.3:0/2815884575 >> v2:172.21.15.92:6800/1750155735 conn(0x7fdfe806c840 0x7fdfe806ed00 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 1 --2- 172.21.15.3:0/2815884575 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe008142e40 0x7fe00813f490 unknown :-1 s=CLOSED pgs=5329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 1 --2- 172.21.15.3:0/2815884575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080ac210 0x7fe00813c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 1 --2- 172.21.15.3:0/2815884575 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe008070480 0x7fe00813d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 1 -- 172.21.15.3:0/2815884575 >> 172.21.15.3:0/2815884575 conn(0x7fe0080afb80 msgr2=0x7fe0081409d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:21.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 1 -- 172.21.15.3:0/2815884575 shutdown_connections 2023-10-30T12:33:21.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 1 -- 172.21.15.3:0/2815884575 wait complete. 2023-10-30T12:33:21.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.428+0000 7fe00d98a640 1 librados: shutdown 2023-10-30T12:33:21.441 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1603,"stamp":"2023-10-30T12:33:20.116678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":578544,"kb_used_data":52976,"kb_used_omap":12,"kb_used_meta":525491,"kb_avail":754396176,"statfs":{"total":773094113280,"available":772501684224,"internally_reserved":0,"allocated":54247424,"data_stored":26382970,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12743,"internal_metadata":538103353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002759"},"pg_stats":[{"pgid":"24.4","version":"0'0","reported_seq":863,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:58.845151+0000","last_change":"2023-10-30T12:32:58.849370+0000","last_active":"2023-10-30T12:32:58.845151+0000","last_peered":"2023-10-30T12:32:58.845151+0000","last_clean":"2023-10-30T12:32:58.845151+0000","last_became_active":"2023-10-30T12:11:50.811294+0000","last_became_peered":"2023-10-30T12:11:50.811294+0000","last_unstale":"2023-10-30T12:32:58.845151+0000","last_undegraded":"2023-10-30T12:32:58.845151+0000","last_fullsized":"2023-10-30T12:32:58.845151+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:58.843751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:58.843751+0000","last_clean_scrub_stamp":"2023-10-30T12:32:58.843751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:13.308129+0000","scrub_duration":0.031572250000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":894,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:47.592582+0000","last_change":"2023-10-30T12:32:47.592582+0000","last_active":"2023-10-30T12:32:47.592582+0000","last_peered":"2023-10-30T12:32:47.592582+0000","last_clean":"2023-10-30T12:32:47.592582+0000","last_became_active":"2023-10-30T12:11:58.558569+0000","last_became_peered":"2023-10-30T12:11:58.558569+0000","last_unstale":"2023-10-30T12:32:47.592582+0000","last_undegraded":"2023-10-30T12:32:47.592582+0000","last_fullsized":"2023-10-30T12:32:47.592582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:47.590178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:47.590178+0000","last_clean_scrub_stamp":"2023-10-30T12:32:47.590178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:11:38.145895+0000","scrub_duration":0.046353002999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":893,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:48.584463+0000","last_change":"2023-10-30T12:32:48.586650+0000","last_active":"2023-10-30T12:32:48.584463+0000","last_peered":"2023-10-30T12:32:48.584463+0000","last_clean":"2023-10-30T12:32:48.584463+0000","last_became_active":"2023-10-30T12:11:58.495252+0000","last_became_peered":"2023-10-30T12:11:58.495252+0000","last_unstale":"2023-10-30T12:32:48.584463+0000","last_undegraded":"2023-10-30T12:32:48.584463+0000","last_fullsized":"2023-10-30T12:32:48.584463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:48.580750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:48.580750+0000","last_clean_scrub_stamp":"2023-10-30T12:32:48.580750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:38:33.376709+0000","scrub_duration":0.0492234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:33:02.984106+0000","last_change":"2023-10-30T12:33:02.985159+0000","last_active":"2023-10-30T12:33:02.984106+0000","last_peered":"2023-10-30T12:33:02.984106+0000","last_clean":"2023-10-30T12:33:02.984106+0000","last_became_active":"2023-10-30T12:12:25.941872+0000","last_became_peered":"2023-10-30T12:12:25.941872+0000","last_unstale":"2023-10-30T12:33:02.984106+0000","last_undegraded":"2023-10-30T12:33:02.984106+0000","last_fullsized":"2023-10-30T12:33:02.984106+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:33:02.982764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:33:02.982764+0000","last_clean_scrub_stamp":"2023-10-30T12:33:02.982764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:49:23.724841+0000","scrub_duration":0.064482645000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":880,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:43.079860+0000","last_change":"2023-10-30T12:32:43.080900+0000","last_active":"2023-10-30T12:32:43.079860+0000","last_peered":"2023-10-30T12:32:43.079860+0000","last_clean":"2023-10-30T12:32:43.079860+0000","last_became_active":"2023-10-30T12:11:51.581699+0000","last_became_peered":"2023-10-30T12:11:51.581699+0000","last_unstale":"2023-10-30T12:32:43.079860+0000","last_undegraded":"2023-10-30T12:32:43.079860+0000","last_fullsized":"2023-10-30T12:32:43.079860+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:43.078454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:43.078454+0000","last_clean_scrub_stamp":"2023-10-30T12:32:43.078454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:22:47.051220+0000","scrub_duration":0.042524077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1","version":"900'8","reported_seq":878,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:44.206487+0000","last_change":"2023-10-30T12:32:44.207698+0000","last_active":"2023-10-30T12:32:44.206487+0000","last_peered":"2023-10-30T12:32:44.206487+0000","last_clean":"2023-10-30T12:32:44.206487+0000","last_became_active":"2023-10-30T12:12:02.320185+0000","last_became_peered":"2023-10-30T12:12:02.320185+0000","last_unstale":"2023-10-30T12:32:44.206487+0000","last_undegraded":"2023-10-30T12:32:44.206487+0000","last_fullsized":"2023-10-30T12:32:44.206487+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"900'8","last_scrub_stamp":"2023-10-30T12:32:44.205034+0000","last_deep_scrub":"900'8","last_deep_scrub_stamp":"2023-10-30T12:32:44.205034+0000","last_clean_scrub_stamp":"2023-10-30T12:32:44.205034+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:31:51.379361+0000","scrub_duration":0.092655721999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":864,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:45.149747+0000","last_change":"2023-10-30T12:32:45.150862+0000","last_active":"2023-10-30T12:32:45.149747+0000","last_peered":"2023-10-30T12:32:45.149747+0000","last_clean":"2023-10-30T12:32:45.149747+0000","last_became_active":"2023-10-30T12:12:02.269307+0000","last_became_peered":"2023-10-30T12:12:02.269307+0000","last_unstale":"2023-10-30T12:32:45.149747+0000","last_undegraded":"2023-10-30T12:32:45.149747+0000","last_fullsized":"2023-10-30T12:32:45.149747+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:45.148252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:45.148252+0000","last_clean_scrub_stamp":"2023-10-30T12:32:45.148252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:36:41.473826+0000","scrub_duration":0.050301950999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":864,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:40.067739+0000","last_change":"2023-10-30T12:32:40.067739+0000","last_active":"2023-10-30T12:32:40.067739+0000","last_peered":"2023-10-30T12:32:40.067739+0000","last_clean":"2023-10-30T12:32:40.067739+0000","last_became_active":"2023-10-30T12:11:55.581296+0000","last_became_peered":"2023-10-30T12:11:55.581296+0000","last_unstale":"2023-10-30T12:32:40.067739+0000","last_undegraded":"2023-10-30T12:32:40.067739+0000","last_fullsized":"2023-10-30T12:32:40.067739+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:40.063585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:40.063585+0000","last_clean_scrub_stamp":"2023-10-30T12:32:40.063585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:43.615070+0000","scrub_duration":0.072154391999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:53.883296+0000","last_change":"2023-10-30T12:32:53.888028+0000","last_active":"2023-10-30T12:32:53.883296+0000","last_peered":"2023-10-30T12:32:53.883296+0000","last_clean":"2023-10-30T12:32:53.883296+0000","last_became_active":"2023-10-30T12:12:25.833774+0000","last_became_peered":"2023-10-30T12:12:25.833774+0000","last_unstale":"2023-10-30T12:32:53.883296+0000","last_undegraded":"2023-10-30T12:32:53.883296+0000","last_fullsized":"2023-10-30T12:32:53.883296+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:53.881823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:53.881823+0000","last_clean_scrub_stamp":"2023-10-30T12:32:53.881823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:47:51.656882+0000","scrub_duration":0.054464448999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:52.866011+0000","last_change":"2023-10-30T12:32:52.867084+0000","last_active":"2023-10-30T12:32:52.866011+0000","last_peered":"2023-10-30T12:32:52.866011+0000","last_clean":"2023-10-30T12:32:52.866011+0000","last_became_active":"2023-10-30T12:12:09.094001+0000","last_became_peered":"2023-10-30T12:12:09.094001+0000","last_unstale":"2023-10-30T12:32:52.866011+0000","last_undegraded":"2023-10-30T12:32:52.866011+0000","last_fullsized":"2023-10-30T12:32:52.866011+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:52.864567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:52.864567+0000","last_clean_scrub_stamp":"2023-10-30T12:32:52.864567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:41:01.102396+0000","scrub_duration":0.062358990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":878,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:50.125424+0000","last_change":"2023-10-30T12:32:50.129322+0000","last_active":"2023-10-30T12:32:50.125424+0000","last_peered":"2023-10-30T12:32:50.125424+0000","last_clean":"2023-10-30T12:32:50.125424+0000","last_became_active":"2023-10-30T12:12:04.077824+0000","last_became_peered":"2023-10-30T12:12:04.077824+0000","last_unstale":"2023-10-30T12:32:50.125424+0000","last_undegraded":"2023-10-30T12:32:50.125424+0000","last_fullsized":"2023-10-30T12:32:50.125424+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:50.124037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:50.124037+0000","last_clean_scrub_stamp":"2023-10-30T12:32:50.124037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:43:02.338696+0000","scrub_duration":0.101601231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":860,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:33:07.826660+0000","last_change":"2023-10-30T12:33:07.827691+0000","last_active":"2023-10-30T12:33:07.826660+0000","last_peered":"2023-10-30T12:33:07.826660+0000","last_clean":"2023-10-30T12:33:07.826660+0000","last_became_active":"2023-10-30T12:11:50.987299+0000","last_became_peered":"2023-10-30T12:11:50.987299+0000","last_unstale":"2023-10-30T12:33:07.826660+0000","last_undegraded":"2023-10-30T12:33:07.826660+0000","last_fullsized":"2023-10-30T12:33:07.826660+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:33:07.825270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:33:07.825270+0000","last_clean_scrub_stamp":"2023-10-30T12:33:07.825270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:25.653873+0000","scrub_duration":0.064363538999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":861,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:33:04.864784+0000","last_change":"2023-10-30T12:33:04.865841+0000","last_active":"2023-10-30T12:33:04.864784+0000","last_peered":"2023-10-30T12:33:04.864784+0000","last_clean":"2023-10-30T12:33:04.864784+0000","last_became_active":"2023-10-30T12:11:51.023075+0000","last_became_peered":"2023-10-30T12:11:51.023075+0000","last_unstale":"2023-10-30T12:33:04.864784+0000","last_undegraded":"2023-10-30T12:33:04.864784+0000","last_fullsized":"2023-10-30T12:33:04.864784+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:33:04.863369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:33:04.863369+0000","last_clean_scrub_stamp":"2023-10-30T12:33:04.863369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:43.227099+0000","scrub_duration":0.052852314999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":884,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:41.613858+0000","last_change":"2023-10-30T12:32:41.614961+0000","last_active":"2023-10-30T12:32:41.613858+0000","last_peered":"2023-10-30T12:32:41.613858+0000","last_clean":"2023-10-30T12:32:41.613858+0000","last_became_active":"2023-10-30T12:10:17.738241+0000","last_became_peered":"2023-10-30T12:10:17.738241+0000","last_unstale":"2023-10-30T12:32:41.613858+0000","last_undegraded":"2023-10-30T12:32:41.613858+0000","last_fullsized":"2023-10-30T12:32:41.613858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:41.612385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:41.612385+0000","last_clean_scrub_stamp":"2023-10-30T12:32:41.612385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:18:13.401922+0000","scrub_duration":0.035484615999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":870,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:39.895283+0000","last_change":"2023-10-30T12:32:39.896413+0000","last_active":"2023-10-30T12:32:39.895283+0000","last_peered":"2023-10-30T12:32:39.895283+0000","last_clean":"2023-10-30T12:32:39.895283+0000","last_became_active":"2023-10-30T12:11:52.257573+0000","last_became_peered":"2023-10-30T12:11:52.257573+0000","last_unstale":"2023-10-30T12:32:39.895283+0000","last_undegraded":"2023-10-30T12:32:39.895283+0000","last_fullsized":"2023-10-30T12:32:39.895283+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:39.893840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:39.893840+0000","last_clean_scrub_stamp":"2023-10-30T12:32:39.893840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:04.201929+0000","scrub_duration":0.031836309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":885,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:40.115821+0000","last_change":"2023-10-30T12:32:40.116902+0000","last_active":"2023-10-30T12:32:40.115821+0000","last_peered":"2023-10-30T12:32:40.115821+0000","last_clean":"2023-10-30T12:32:40.115821+0000","last_became_active":"2023-10-30T12:10:17.763227+0000","last_became_peered":"2023-10-30T12:10:17.763227+0000","last_unstale":"2023-10-30T12:32:40.115821+0000","last_undegraded":"2023-10-30T12:32:40.115821+0000","last_fullsized":"2023-10-30T12:32:40.115821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:40.114411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:40.114411+0000","last_clean_scrub_stamp":"2023-10-30T12:32:40.114411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:05.152002+0000","scrub_duration":0.037475106000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":894,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:50.536434+0000","last_change":"2023-10-30T12:32:50.537406+0000","last_active":"2023-10-30T12:32:50.536434+0000","last_peered":"2023-10-30T12:32:50.536434+0000","last_clean":"2023-10-30T12:32:50.536434+0000","last_became_active":"2023-10-30T12:11:55.613173+0000","last_became_peered":"2023-10-30T12:11:55.613173+0000","last_unstale":"2023-10-30T12:32:50.536434+0000","last_undegraded":"2023-10-30T12:32:50.536434+0000","last_fullsized":"2023-10-30T12:32:50.536434+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:50.535095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:50.535095+0000","last_clean_scrub_stamp":"2023-10-30T12:32:50.535095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:14.593021+0000","scrub_duration":0.061830881999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":884,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:44.659407+0000","last_change":"2023-10-30T12:32:44.660493+0000","last_active":"2023-10-30T12:32:44.659407+0000","last_peered":"2023-10-30T12:32:44.659407+0000","last_clean":"2023-10-30T12:32:44.659407+0000","last_became_active":"2023-10-30T12:10:16.907205+0000","last_became_peered":"2023-10-30T12:10:16.907205+0000","last_unstale":"2023-10-30T12:32:44.659407+0000","last_undegraded":"2023-10-30T12:32:44.659407+0000","last_fullsized":"2023-10-30T12:32:44.659407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:44.658017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:44.658017+0000","last_clean_scrub_stamp":"2023-10-30T12:32:44.658017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:04.891705+0000","scrub_duration":0.033505479999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":871,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:37.904809+0000","last_change":"2023-10-30T12:32:37.905916+0000","last_active":"2023-10-30T12:32:37.904809+0000","last_peered":"2023-10-30T12:32:37.904809+0000","last_clean":"2023-10-30T12:32:37.904809+0000","last_became_active":"2023-10-30T12:11:52.909795+0000","last_became_peered":"2023-10-30T12:11:52.909795+0000","last_unstale":"2023-10-30T12:32:37.904809+0000","last_undegraded":"2023-10-30T12:32:37.904809+0000","last_fullsized":"2023-10-30T12:32:37.904809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:37.903361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:37.903361+0000","last_clean_scrub_stamp":"2023-10-30T12:32:37.903361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:22.724580+0000","scrub_duration":0.049141183999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":900,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:38.541118+0000","last_change":"2023-10-30T12:32:38.545483+0000","last_active":"2023-10-30T12:32:38.541118+0000","last_peered":"2023-10-30T12:32:38.541118+0000","last_clean":"2023-10-30T12:32:38.541118+0000","last_became_active":"2023-10-30T12:10:17.582475+0000","last_became_peered":"2023-10-30T12:10:17.582475+0000","last_unstale":"2023-10-30T12:32:38.541118+0000","last_undegraded":"2023-10-30T12:32:38.541118+0000","last_fullsized":"2023-10-30T12:32:38.541118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:38.539673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:38.539673+0000","last_clean_scrub_stamp":"2023-10-30T12:32:38.539673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:59:45.700659+0000","scrub_duration":0.055551556000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":869,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:39.027151+0000","last_change":"2023-10-30T12:32:39.029970+0000","last_active":"2023-10-30T12:32:39.027151+0000","last_peered":"2023-10-30T12:32:39.027151+0000","last_clean":"2023-10-30T12:32:39.027151+0000","last_became_active":"2023-10-30T12:11:51.581905+0000","last_became_peered":"2023-10-30T12:11:51.581905+0000","last_unstale":"2023-10-30T12:32:39.027151+0000","last_undegraded":"2023-10-30T12:32:39.027151+0000","last_fullsized":"2023-10-30T12:32:39.027151+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:39.022101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:39.022101+0000","last_clean_scrub_stamp":"2023-10-30T12:32:39.022101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:10:34.443820+0000","scrub_duration":0.045951260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":885,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:42.576311+0000","last_change":"2023-10-30T12:32:42.576311+0000","last_active":"2023-10-30T12:32:42.576311+0000","last_peered":"2023-10-30T12:32:42.576311+0000","last_clean":"2023-10-30T12:32:42.576311+0000","last_became_active":"2023-10-30T12:10:17.553523+0000","last_became_peered":"2023-10-30T12:10:17.553523+0000","last_unstale":"2023-10-30T12:32:42.576311+0000","last_undegraded":"2023-10-30T12:32:42.576311+0000","last_fullsized":"2023-10-30T12:32:42.576311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:42.573746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:42.573746+0000","last_clean_scrub_stamp":"2023-10-30T12:32:42.573746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:38.805371+0000","scrub_duration":0.038013714999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":861,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:33:03.878503+0000","last_change":"2023-10-30T12:33:03.879496+0000","last_active":"2023-10-30T12:33:03.878503+0000","last_peered":"2023-10-30T12:33:03.878503+0000","last_clean":"2023-10-30T12:33:03.878503+0000","last_became_active":"2023-10-30T12:11:50.965758+0000","last_became_peered":"2023-10-30T12:11:50.965758+0000","last_unstale":"2023-10-30T12:33:03.878503+0000","last_undegraded":"2023-10-30T12:33:03.878503+0000","last_fullsized":"2023-10-30T12:33:03.878503+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:33:03.877167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:33:03.877167+0000","last_clean_scrub_stamp":"2023-10-30T12:33:03.877167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:35:35.618691+0000","scrub_duration":0.057181607000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":870,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:43.939195+0000","last_change":"2023-10-30T12:32:43.939195+0000","last_active":"2023-10-30T12:32:43.939195+0000","last_peered":"2023-10-30T12:32:43.939195+0000","last_clean":"2023-10-30T12:32:43.939195+0000","last_became_active":"2023-10-30T12:10:16.811711+0000","last_became_peered":"2023-10-30T12:10:16.811711+0000","last_unstale":"2023-10-30T12:32:43.939195+0000","last_undegraded":"2023-10-30T12:32:43.939195+0000","last_fullsized":"2023-10-30T12:32:43.939195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:43.935642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:43.935642+0000","last_clean_scrub_stamp":"2023-10-30T12:32:43.935642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:19:24.462349+0000","scrub_duration":0.034953691000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":861,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:56.861599+0000","last_change":"2023-10-30T12:32:56.862623+0000","last_active":"2023-10-30T12:32:56.861599+0000","last_peered":"2023-10-30T12:32:56.861599+0000","last_clean":"2023-10-30T12:32:56.861599+0000","last_became_active":"2023-10-30T12:12:02.408646+0000","last_became_peered":"2023-10-30T12:12:02.408646+0000","last_unstale":"2023-10-30T12:32:56.861599+0000","last_undegraded":"2023-10-30T12:32:56.861599+0000","last_fullsized":"2023-10-30T12:32:56.861599+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:56.860189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:56.860189+0000","last_clean_scrub_stamp":"2023-10-30T12:32:56.860189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:19.653177+0000","scrub_duration":0.040718254000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":885,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:36.670764+0000","last_change":"2023-10-30T12:32:36.671858+0000","last_active":"2023-10-30T12:32:36.670764+0000","last_peered":"2023-10-30T12:32:36.670764+0000","last_clean":"2023-10-30T12:32:36.670764+0000","last_became_active":"2023-10-30T12:10:16.849940+0000","last_became_peered":"2023-10-30T12:10:16.849940+0000","last_unstale":"2023-10-30T12:32:36.670764+0000","last_undegraded":"2023-10-30T12:32:36.670764+0000","last_fullsized":"2023-10-30T12:32:36.670764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:36.669332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:36.669332+0000","last_clean_scrub_stamp":"2023-10-30T12:32:36.669332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:53:28.553745+0000","scrub_duration":0.090791832000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":870,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:36.882845+0000","last_change":"2023-10-30T12:32:36.883963+0000","last_active":"2023-10-30T12:32:36.882845+0000","last_peered":"2023-10-30T12:32:36.882845+0000","last_clean":"2023-10-30T12:32:36.882845+0000","last_became_active":"2023-10-30T12:11:53.066917+0000","last_became_peered":"2023-10-30T12:11:53.066917+0000","last_unstale":"2023-10-30T12:32:36.882845+0000","last_undegraded":"2023-10-30T12:32:36.882845+0000","last_fullsized":"2023-10-30T12:32:36.882845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:36.881387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:36.881387+0000","last_clean_scrub_stamp":"2023-10-30T12:32:36.881387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:33:50.617590+0000","scrub_duration":0.073083355000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":885,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:38.640066+0000","last_change":"2023-10-30T12:32:38.640066+0000","last_active":"2023-10-30T12:32:38.640066+0000","last_peered":"2023-10-30T12:32:38.640066+0000","last_clean":"2023-10-30T12:32:38.640066+0000","last_became_active":"2023-10-30T12:10:17.153120+0000","last_became_peered":"2023-10-30T12:10:17.153120+0000","last_unstale":"2023-10-30T12:32:38.640066+0000","last_undegraded":"2023-10-30T12:32:38.640066+0000","last_fullsized":"2023-10-30T12:32:38.640066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:38.636075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:38.636075+0000","last_clean_scrub_stamp":"2023-10-30T12:32:38.636075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:06:44.125261+0000","scrub_duration":0.047870970999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"900'78","reported_seq":979,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:39.080995+0000","last_change":"2023-10-30T12:32:39.085226+0000","last_active":"2023-10-30T12:32:39.080995+0000","last_peered":"2023-10-30T12:32:39.080995+0000","last_clean":"2023-10-30T12:32:39.080995+0000","last_became_active":"2023-10-30T12:10:09.748373+0000","last_became_peered":"2023-10-30T12:10:09.748373+0000","last_unstale":"2023-10-30T12:32:39.080995+0000","last_undegraded":"2023-10-30T12:32:39.080995+0000","last_fullsized":"2023-10-30T12:32:39.080995+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"900'78","last_scrub_stamp":"2023-10-30T12:32:39.079215+0000","last_deep_scrub":"900'78","last_deep_scrub_stamp":"2023-10-30T12:32:39.079215+0000","last_clean_scrub_stamp":"2023-10-30T12:32:39.079215+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:31.600901+0000","scrub_duration":0.127691637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":871,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:50.920526+0000","last_change":"2023-10-30T12:32:50.920526+0000","last_active":"2023-10-30T12:32:50.920526+0000","last_peered":"2023-10-30T12:32:50.920526+0000","last_clean":"2023-10-30T12:32:50.920526+0000","last_became_active":"2023-10-30T12:12:03.631410+0000","last_became_peered":"2023-10-30T12:12:03.631410+0000","last_unstale":"2023-10-30T12:32:50.920526+0000","last_undegraded":"2023-10-30T12:32:50.920526+0000","last_fullsized":"2023-10-30T12:32:50.920526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:50.918001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:50.918001+0000","last_clean_scrub_stamp":"2023-10-30T12:32:50.918001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:09:50.307402+0000","scrub_duration":0.056243267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":880,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:40.819762+0000","last_change":"2023-10-30T12:32:40.819762+0000","last_active":"2023-10-30T12:32:40.819762+0000","last_peered":"2023-10-30T12:32:40.819762+0000","last_clean":"2023-10-30T12:32:40.819762+0000","last_became_active":"2023-10-30T12:12:01.926913+0000","last_became_peered":"2023-10-30T12:12:01.926913+0000","last_unstale":"2023-10-30T12:32:40.819762+0000","last_undegraded":"2023-10-30T12:32:40.819762+0000","last_fullsized":"2023-10-30T12:32:40.819762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:40.814697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:40.814697+0000","last_clean_scrub_stamp":"2023-10-30T12:32:40.814697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:57.003617+0000","scrub_duration":0.028773494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":862,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:46.972772+0000","last_change":"2023-10-30T12:32:46.972772+0000","last_active":"2023-10-30T12:32:46.972772+0000","last_peered":"2023-10-30T12:32:46.972772+0000","last_clean":"2023-10-30T12:32:46.972772+0000","last_became_active":"2023-10-30T12:12:02.108733+0000","last_became_peered":"2023-10-30T12:12:02.108733+0000","last_unstale":"2023-10-30T12:32:46.972772+0000","last_undegraded":"2023-10-30T12:32:46.972772+0000","last_fullsized":"2023-10-30T12:32:46.972772+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:46.970477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:46.970477+0000","last_clean_scrub_stamp":"2023-10-30T12:32:46.970477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:51:24.619904+0000","scrub_duration":0.038933902999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":870,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:36.941395+0000","last_change":"2023-10-30T12:32:36.941395+0000","last_active":"2023-10-30T12:32:36.941395+0000","last_peered":"2023-10-30T12:32:36.941395+0000","last_clean":"2023-10-30T12:32:36.941395+0000","last_became_active":"2023-10-30T12:11:52.304910+0000","last_became_peered":"2023-10-30T12:11:52.304910+0000","last_unstale":"2023-10-30T12:32:36.941395+0000","last_undegraded":"2023-10-30T12:32:36.941395+0000","last_fullsized":"2023-10-30T12:32:36.941395+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:36.936947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:36.936947+0000","last_clean_scrub_stamp":"2023-10-30T12:32:36.936947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:37:08.108230+0000","scrub_duration":0.035594848999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":876,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:58.934905+0000","last_change":"2023-10-30T12:32:58.935990+0000","last_active":"2023-10-30T12:32:58.934905+0000","last_peered":"2023-10-30T12:32:58.934905+0000","last_clean":"2023-10-30T12:32:58.934905+0000","last_became_active":"2023-10-30T12:12:10.791950+0000","last_became_peered":"2023-10-30T12:12:10.791950+0000","last_unstale":"2023-10-30T12:32:58.934905+0000","last_undegraded":"2023-10-30T12:32:58.934905+0000","last_fullsized":"2023-10-30T12:32:58.934905+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:58.933486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:58.933486+0000","last_clean_scrub_stamp":"2023-10-30T12:32:58.933486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:12:18.423964+0000","scrub_duration":0.068492327000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":879,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:41.850865+0000","last_change":"2023-10-30T12:32:41.851869+0000","last_active":"2023-10-30T12:32:41.850865+0000","last_peered":"2023-10-30T12:32:41.850865+0000","last_clean":"2023-10-30T12:32:41.850865+0000","last_became_active":"2023-10-30T12:11:56.049790+0000","last_became_peered":"2023-10-30T12:11:56.049790+0000","last_unstale":"2023-10-30T12:32:41.850865+0000","last_undegraded":"2023-10-30T12:32:41.850865+0000","last_fullsized":"2023-10-30T12:32:41.850865+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:41.849413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:41.849413+0000","last_clean_scrub_stamp":"2023-10-30T12:32:41.849413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:56:56.410162+0000","scrub_duration":0.037757672999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":861,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:33:02.817550+0000","last_change":"2023-10-30T12:33:02.820947+0000","last_active":"2023-10-30T12:33:02.817550+0000","last_peered":"2023-10-30T12:33:02.817550+0000","last_clean":"2023-10-30T12:33:02.817550+0000","last_became_active":"2023-10-30T12:12:02.435431+0000","last_became_peered":"2023-10-30T12:12:02.435431+0000","last_unstale":"2023-10-30T12:33:02.817550+0000","last_undegraded":"2023-10-30T12:33:02.817550+0000","last_fullsized":"2023-10-30T12:33:02.817550+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:33:02.816262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:33:02.816262+0000","last_clean_scrub_stamp":"2023-10-30T12:33:02.816262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:04.554357+0000","scrub_duration":0.041757642999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":862,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:33:00.826743+0000","last_change":"2023-10-30T12:33:00.831139+0000","last_active":"2023-10-30T12:33:00.826743+0000","last_peered":"2023-10-30T12:33:00.826743+0000","last_clean":"2023-10-30T12:33:00.826743+0000","last_became_active":"2023-10-30T12:11:51.289446+0000","last_became_peered":"2023-10-30T12:11:51.289446+0000","last_unstale":"2023-10-30T12:33:00.826743+0000","last_undegraded":"2023-10-30T12:33:00.826743+0000","last_fullsized":"2023-10-30T12:33:00.826743+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:33:00.825449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:33:00.825449+0000","last_clean_scrub_stamp":"2023-10-30T12:33:00.825449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:32.324198+0000","scrub_duration":0.030547641,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":862,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:33:05.843477+0000","last_change":"2023-10-30T12:33:05.844538+0000","last_active":"2023-10-30T12:33:05.843477+0000","last_peered":"2023-10-30T12:33:05.843477+0000","last_clean":"2023-10-30T12:33:05.843477+0000","last_became_active":"2023-10-30T12:11:53.110390+0000","last_became_peered":"2023-10-30T12:11:53.110390+0000","last_unstale":"2023-10-30T12:33:05.843477+0000","last_undegraded":"2023-10-30T12:33:05.843477+0000","last_fullsized":"2023-10-30T12:33:05.843477+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:33:05.842149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:33:05.842149+0000","last_clean_scrub_stamp":"2023-10-30T12:33:05.842149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:59:03.871727+0000","scrub_duration":0.052813305999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":878,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:44.117738+0000","last_change":"2023-10-30T12:32:44.118805+0000","last_active":"2023-10-30T12:32:44.117738+0000","last_peered":"2023-10-30T12:32:44.117738+0000","last_clean":"2023-10-30T12:32:44.117738+0000","last_became_active":"2023-10-30T12:12:00.784060+0000","last_became_peered":"2023-10-30T12:12:00.784060+0000","last_unstale":"2023-10-30T12:32:44.117738+0000","last_undegraded":"2023-10-30T12:32:44.117738+0000","last_fullsized":"2023-10-30T12:32:44.117738+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:44.116345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:44.116345+0000","last_clean_scrub_stamp":"2023-10-30T12:32:44.116345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:01:13.298323+0000","scrub_duration":0.041930035999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":878,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:45.123387+0000","last_change":"2023-10-30T12:32:45.124352+0000","last_active":"2023-10-30T12:32:45.123387+0000","last_peered":"2023-10-30T12:32:45.123387+0000","last_clean":"2023-10-30T12:32:45.123387+0000","last_became_active":"2023-10-30T12:12:01.679359+0000","last_became_peered":"2023-10-30T12:12:01.679359+0000","last_unstale":"2023-10-30T12:32:45.123387+0000","last_undegraded":"2023-10-30T12:32:45.123387+0000","last_fullsized":"2023-10-30T12:32:45.123387+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:45.121746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:45.121746+0000","last_clean_scrub_stamp":"2023-10-30T12:32:45.121746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:47:20.330160+0000","scrub_duration":0.048729416999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":877,"reported_epoch":900,"state":"active+clean","last_fresh":"2023-10-30T12:32:50.798358+0000","last_change":"2023-10-30T12:32:50.799469+0000","last_active":"2023-10-30T12:32:50.798358+0000","last_peered":"2023-10-30T12:32:50.798358+0000","last_clean":"2023-10-30T12:32:50.798358+0000","last_became_active":"2023-10-30T12:12:08.259922+0000","last_became_peered":"2023-10-30T12:12:08.259922+0000","last_unstale":"2023-10-30T12:32:50.798358+0000","last_undegraded":"2023-10-30T12:32:50.798358+0000","last_fullsized":"2023-10-30T12:32:50.798358+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:32:50.794996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:32:50.794996+0000","last_clean_scrub_stamp":"2023-10-30T12:32:50.794996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:00.183382+0000","scrub_duration":0.04023127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":4496,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477017,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78044,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293796,"statfs":{"total":96636764160,"available":96556847104,"internally_reserved":0,"allocated":6770688,"data_stored":3198410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73136584},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:32:57 2023","interfaces":[{"interface":"back","average":{"1min":10.19,"5min":15.43,"15min":18.722999999999999},"min":{"1min":5.9960000000000004,"5min":5.2220000000000004,"15min":4.1319999999999997},"max":{"1min":27.266999999999999,"5min":120.447,"15min":213.76499999999999},"last":7.1669999999999998},{"interface":"front","average":{"1min":14.603999999999999,"5min":17.568000000000001,"15min":22.356000000000002},"min":{"1min":5.1699999999999999,"5min":4.6660000000000004,"15min":4.1360000000000001},"max":{"1min":39.094999999999999,"5min":130.48699999999999,"15min":255.06999999999999},"last":6.266}]},{"osd":1,"last update":"Mon Oct 30 12:32:59 2023","interfaces":[{"interface":"back","average":{"1min":13.378,"5min":18.187000000000001,"15min":20.876999999999999},"min":{"1min":5.6660000000000004,"5min":5.3449999999999998,"15min":4.6609999999999996},"max":{"1min":105.25700000000001,"5min":118.071,"15min":164.755},"last":8.2550000000000008},{"interface":"front","average":{"1min":13.916,"5min":17.806999999999999,"15min":23.449000000000002},"min":{"1min":6.2389999999999999,"5min":6.2389999999999999,"15min":6.0990000000000002},"max":{"1min":50.499000000000002,"5min":101.54600000000001,"15min":182.20599999999999},"last":12.598000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:32:57 2023","interfaces":[{"interface":"back","average":{"1min":15.801,"5min":22.035,"15min":30.111000000000001},"min":{"1min":8.4570000000000007,"5min":7.1929999999999996,"15min":6.4589999999999996},"max":{"1min":40.378,"5min":215.48400000000001,"15min":234.21299999999999},"last":11.962999999999999},{"interface":"front","average":{"1min":15.904,"5min":22.300000000000001,"15min":29.846},"min":{"1min":8.4260000000000002,"5min":6.5860000000000003,"15min":6.5860000000000003},"max":{"1min":45.652000000000001,"5min":254.81999999999999,"15min":254.81999999999999},"last":14.788}]},{"osd":4,"last update":"Mon Oct 30 12:32:57 2023","interfaces":[{"interface":"back","average":{"1min":23.488,"5min":20.946000000000002,"15min":28.969999999999999},"min":{"1min":9.0549999999999997,"5min":8.7599999999999998,"15min":7.8780000000000001},"max":{"1min":203.12100000000001,"5min":203.12100000000001,"15min":287.06200000000001},"last":15.762},{"interface":"front","average":{"1min":26.443000000000001,"5min":23.152999999999999,"15min":32.545999999999999},"min":{"1min":13.247999999999999,"5min":11.537000000000001,"15min":10.587},"max":{"1min":188.25299999999999,"5min":188.25299999999999,"15min":259.279},"last":19.257999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:32:59 2023","interfaces":[{"interface":"back","average":{"1min":22.550999999999998,"5min":26.571000000000002,"15min":32.911000000000001},"min":{"1min":11.368,"5min":11.368,"15min":8.2550000000000008},"max":{"1min":114.733,"5min":324.26600000000002,"15min":324.26600000000002},"last":16.593},{"interface":"front","average":{"1min":15.837,"5min":18.446000000000002,"15min":22.029},"min":{"1min":8.9100000000000001,"5min":6.6900000000000004,"15min":6.5780000000000003},"max":{"1min":58.869999999999997,"5min":251.57900000000001,"15min":251.57900000000001},"last":9.1660000000000004}]},{"osd":6,"last update":"Mon Oct 30 12:32:59 2023","interfaces":[{"interface":"back","average":{"1min":19.277999999999999,"5min":25.295000000000002,"15min":26.788},"min":{"1min":10.871,"5min":10.584,"15min":7.9370000000000003},"max":{"1min":45.164999999999999,"5min":121.20999999999999,"15min":218.80600000000001},"last":17.542999999999999},{"interface":"front","average":{"1min":15.343,"5min":17.942,"15min":25.277999999999999},"min":{"1min":9.5340000000000007,"5min":8.8859999999999992,"15min":6.8170000000000002},"max":{"1min":49.427999999999997,"5min":121.717,"15min":239.80500000000001},"last":9.8960000000000008}]},{"osd":7,"last update":"Mon Oct 30 12:32:59 2023","interfaces":[{"interface":"back","average":{"1min":21.539999999999999,"5min":30.940000000000001,"15min":38.909999999999997},"min":{"1min":12.068,"5min":12.068,"15min":10.132999999999999},"max":{"1min":54.213999999999999,"5min":323.58199999999999,"15min":1077.606},"last":18.199999999999999},{"interface":"front","average":{"1min":16.812999999999999,"5min":23.981000000000002,"15min":27.151},"min":{"1min":9.484,"5min":9.3049999999999997,"15min":8.5879999999999992},"max":{"1min":48.273000000000003,"5min":253.04599999999999,"15min":253.04599999999999},"last":10.513}]}]},{"osd":2,"up_from":16,"seq":68719477016,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60000,"kb_used_data":6616,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311840,"statfs":{"total":96636764160,"available":96575324160,"internally_reserved":0,"allocated":6774784,"data_stored":3198415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:32:52 2023","interfaces":[{"interface":"back","average":{"1min":13.906000000000001,"5min":16.802,"15min":20.882000000000001},"min":{"1min":6.21,"5min":4.5149999999999997,"15min":4.5149999999999997},"max":{"1min":81.200000000000003,"5min":205.785,"15min":398.01499999999999},"last":16.181000000000001},{"interface":"front","average":{"1min":10.271000000000001,"5min":17.238,"15min":20.475000000000001},"min":{"1min":5.851,"5min":5.851,"15min":4.4790000000000001},"max":{"1min":18.594000000000001,"5min":135.32499999999999,"15min":376.49299999999999},"last":8.6709999999999994}]},{"osd":1,"last update":"Mon Oct 30 12:32:52 2023","interfaces":[{"interface":"back","average":{"1min":12.113,"5min":17.210999999999999,"15min":22.742000000000001},"min":{"1min":8.2620000000000005,"5min":7.2519999999999998,"15min":6.9660000000000002},"max":{"1min":20.853999999999999,"5min":172.09999999999999,"15min":304.97500000000002},"last":10.08},{"interface":"front","average":{"1min":24.577000000000002,"5min":23.436,"15min":29.164000000000001},"min":{"1min":10.419,"5min":9.9280000000000008,"15min":7.9989999999999997},"max":{"1min":181.94900000000001,"5min":183.803,"15min":355.99400000000003},"last":17.417000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:32:52 2023","interfaces":[{"interface":"back","average":{"1min":16.068000000000001,"5min":22.236999999999998,"15min":25.489000000000001},"min":{"1min":8.7219999999999995,"5min":8.7219999999999995,"15min":6.4080000000000004},"max":{"1min":42.259,"5min":207.35300000000001,"15min":412.42899999999997},"last":18.294},{"interface":"front","average":{"1min":12.247,"5min":16.704000000000001,"15min":24.146999999999998},"min":{"1min":7.3929999999999998,"5min":6.1959999999999997,"15min":6.1959999999999997},"max":{"1min":19.873000000000001,"5min":147.63900000000001,"15min":328.90899999999999},"last":10.736000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:32:52 2023","interfaces":[{"interface":"back","average":{"1min":20.837,"5min":17.399000000000001,"15min":20.338999999999999},"min":{"1min":6.9390000000000001,"5min":6.9390000000000001,"15min":6.5439999999999996},"max":{"1min":200.41300000000001,"5min":200.41300000000001,"15min":326.565},"last":11.385999999999999},{"interface":"front","average":{"1min":19.346,"5min":19.966999999999999,"15min":24.449000000000002},"min":{"1min":10.882999999999999,"5min":10.289,"15min":9.8710000000000004},"max":{"1min":45.310000000000002,"5min":83.744,"15min":302.76499999999999},"last":12.113}]},{"osd":5,"last update":"Mon Oct 30 12:32:52 2023","interfaces":[{"interface":"back","average":{"1min":31.780999999999999,"5min":22.683,"15min":21.606999999999999},"min":{"1min":9.7829999999999995,"5min":5.4139999999999997,"15min":4.9539999999999997},"max":{"1min":211.773,"5min":211.773,"15min":246.011},"last":21.943999999999999},{"interface":"front","average":{"1min":17.983000000000001,"5min":15.692,"15min":26.936},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":208.166,"5min":208.166,"15min":350.80399999999997},"last":6.4539999999999997}]},{"osd":6,"last update":"Mon Oct 30 12:32:52 2023","interfaces":[{"interface":"back","average":{"1min":24.526,"5min":22.323,"15min":25.875},"min":{"1min":10.798,"5min":9.5139999999999993,"15min":7.8949999999999996},"max":{"1min":170.02799999999999,"5min":170.02799999999999,"15min":374.16699999999997},"last":12.845000000000001},{"interface":"front","average":{"1min":26.651,"5min":24.942,"15min":28.331},"min":{"1min":9.391,"5min":9.391,"15min":8.2400000000000002},"max":{"1min":184.88999999999999,"5min":184.88999999999999,"15min":333.37299999999999},"last":23.295999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:32:52 2023","interfaces":[{"interface":"back","average":{"1min":12.891,"5min":14.82,"15min":20.096},"min":{"1min":7.6379999999999999,"5min":6.8049999999999997,"15min":6.1539999999999999},"max":{"1min":23.285,"5min":88.486999999999995,"15min":272.11700000000002},"last":14.537000000000001},{"interface":"front","average":{"1min":16.129999999999999,"5min":19.026,"15min":27.291},"min":{"1min":11.071,"5min":9.7449999999999992,"15min":7.4480000000000004},"max":{"1min":32.222000000000001,"5min":114.895,"15min":453.28399999999999},"last":15.292}]}]},{"osd":0,"up_from":16,"seq":68719477016,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59996,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311844,"statfs":{"total":96636764160,"available":96575328256,"internally_reserved":0,"allocated":6770688,"data_stored":3198410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:32:49 2023","interfaces":[{"interface":"back","average":{"1min":17.963999999999999,"5min":19.321999999999999,"15min":18.98},"min":{"1min":7.4240000000000004,"5min":5.6520000000000001,"15min":4.79},"max":{"1min":96.656999999999996,"5min":152.46799999999999,"15min":199.46299999999999},"last":8.1959999999999997},{"interface":"front","average":{"1min":14.843999999999999,"5min":17.559999999999999,"15min":19.123000000000001},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.5279999999999996},"max":{"1min":89.953999999999994,"5min":150.90100000000001,"15min":218.26300000000001},"last":12.643000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:32:49 2023","interfaces":[{"interface":"back","average":{"1min":22.099,"5min":19.155999999999999,"15min":19.678000000000001},"min":{"1min":8.5709999999999997,"5min":5.9580000000000002,"15min":5.9580000000000002},"max":{"1min":102.59699999999999,"5min":120.355,"15min":150.55799999999999},"last":7.4219999999999997},{"interface":"front","average":{"1min":20.047000000000001,"5min":17.795999999999999,"15min":18.312000000000001},"min":{"1min":7.3710000000000004,"5min":6.7439999999999998,"15min":5.8010000000000002},"max":{"1min":92.296000000000006,"5min":101.89100000000001,"15min":144.61099999999999},"last":18.006}]},{"osd":3,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":26.065000000000001,"5min":20.931000000000001,"15min":20.044},"min":{"1min":6.1289999999999996,"5min":5.484,"15min":5.484},"max":{"1min":221.83099999999999,"5min":221.83099999999999,"15min":221.83099999999999},"last":9.6780000000000008},{"interface":"front","average":{"1min":31.369,"5min":21.067,"15min":21.018999999999998},"min":{"1min":9.048,"5min":6.2770000000000001,"15min":5.8129999999999997},"max":{"1min":200.738,"5min":220.589,"15min":220.589},"last":19.667000000000002}]},{"osd":4,"last update":"Mon Oct 30 12:32:49 2023","interfaces":[{"interface":"back","average":{"1min":22.135999999999999,"5min":21.305,"15min":21.172999999999998},"min":{"1min":8.75,"5min":8.75,"15min":6.5949999999999998},"max":{"1min":94.218999999999994,"5min":116.10599999999999,"15min":247.57300000000001},"last":20.562999999999999},{"interface":"front","average":{"1min":17.291,"5min":17.158999999999999,"15min":20.523},"min":{"1min":8.0489999999999995,"5min":6.7370000000000001,"15min":6.7370000000000001},"max":{"1min":80.671999999999997,"5min":115.371,"15min":267.23899999999998},"last":13.782}]},{"osd":5,"last update":"Mon Oct 30 12:32:49 2023","interfaces":[{"interface":"back","average":{"1min":18.238,"5min":19.504000000000001,"15min":21.047999999999998},"min":{"1min":7.6109999999999998,"5min":7.3899999999999997,"15min":6.7080000000000002},"max":{"1min":63.841999999999999,"5min":210.66800000000001,"15min":232.041},"last":8.8640000000000008},{"interface":"front","average":{"1min":23.806000000000001,"5min":23.82,"15min":24.617000000000001},"min":{"1min":12.744,"5min":9.843,"15min":9.843},"max":{"1min":92.659999999999997,"5min":211.376,"15min":256.58699999999999},"last":17.137}]},{"osd":6,"last update":"Mon Oct 30 12:32:49 2023","interfaces":[{"interface":"back","average":{"1min":23.859000000000002,"5min":24.050000000000001,"15min":23.568000000000001},"min":{"1min":10.741,"5min":8.2750000000000004,"15min":8.2750000000000004},"max":{"1min":99.924999999999997,"5min":136.44399999999999,"15min":147.01499999999999},"last":21.829000000000001},{"interface":"front","average":{"1min":23.611999999999998,"5min":24.963000000000001,"15min":24.300000000000001},"min":{"1min":10.833,"5min":8.4619999999999997,"15min":8.4619999999999997},"max":{"1min":71.478999999999999,"5min":196.33799999999999,"15min":196.33799999999999},"last":11.961}]},{"osd":7,"last update":"Mon Oct 30 12:32:49 2023","interfaces":[{"interface":"back","average":{"1min":19.381,"5min":17.393000000000001,"15min":19.437999999999999},"min":{"1min":9.3360000000000003,"5min":7.2110000000000003,"15min":7.2110000000000003},"max":{"1min":90.811000000000007,"5min":95.358999999999995,"15min":180.99100000000001},"last":15.51},{"interface":"front","average":{"1min":26.015000000000001,"5min":23.294,"15min":25.687000000000001},"min":{"1min":13.375999999999999,"5min":9.1110000000000007,"15min":8.827},"max":{"1min":103.71599999999999,"5min":104.599,"15min":199.85300000000001},"last":14.372}]}]},{"osd":7,"up_from":14,"seq":60129542425,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59996,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311844,"statfs":{"total":96636764160,"available":96575328256,"internally_reserved":0,"allocated":6770688,"data_stored":3198410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":13.044,"5min":12.593,"15min":18.446000000000002},"min":{"1min":4.7569999999999997,"5min":4.4199999999999999,"15min":3.7999999999999998},"max":{"1min":119.88,"5min":175.166,"15min":300.30399999999997},"last":15.177},{"interface":"front","average":{"1min":11.448,"5min":11.372999999999999,"15min":20.356999999999999},"min":{"1min":4.0860000000000003,"5min":4.0570000000000004,"15min":4.0570000000000004},"max":{"1min":99.126000000000005,"5min":101.81399999999999,"15min":1113.894},"last":10.231999999999999}]},{"osd":1,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":15.704000000000001,"5min":14.545999999999999,"15min":18.707000000000001},"min":{"1min":6.649,"5min":5.1840000000000002,"15min":4.5839999999999996},"max":{"1min":61.970999999999997,"5min":70.786000000000001,"15min":282.928},"last":11.029999999999999},{"interface":"front","average":{"1min":17.155999999999999,"5min":16.393999999999998,"15min":22.126999999999999},"min":{"1min":7.1319999999999997,"5min":5.524,"15min":4.7300000000000004},"max":{"1min":80.436000000000007,"5min":105.95399999999999,"15min":331.18700000000001},"last":20.859999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":17.294,"5min":16.800999999999998,"15min":23.338999999999999},"min":{"1min":6.3819999999999997,"5min":6.3819999999999997,"15min":5.242},"max":{"1min":56.945999999999998,"5min":96.292000000000002,"15min":309.40600000000001},"last":7.3849999999999998},{"interface":"front","average":{"1min":18.611000000000001,"5min":18.437999999999999,"15min":25.881},"min":{"1min":7.7229999999999999,"5min":6.6970000000000001,"15min":5.883},"max":{"1min":120.937,"5min":172.32300000000001,"15min":318.04000000000002},"last":8.4689999999999994}]},{"osd":3,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":17.614999999999998,"5min":19.256,"15min":26.411000000000001},"min":{"1min":6.8799999999999999,"5min":5.7279999999999998,"15min":5.7279999999999998},"max":{"1min":96.552000000000007,"5min":211.71899999999999,"15min":419.54599999999999},"last":11.643000000000001},{"interface":"front","average":{"1min":19.911999999999999,"5min":22.864000000000001,"15min":27.788},"min":{"1min":9.8499999999999996,"5min":8.0380000000000003,"15min":7.4969999999999999},"max":{"1min":60.283000000000001,"5min":280.57400000000001,"15min":353.90800000000002},"last":9.2110000000000003}]},{"osd":4,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":16.817,"5min":18.376999999999999,"15min":23.847999999999999},"min":{"1min":6.069,"5min":6.069,"15min":5.7039999999999997},"max":{"1min":97.218999999999994,"5min":158.77000000000001,"15min":420.41399999999999},"last":16.271999999999998},{"interface":"front","average":{"1min":17.460000000000001,"5min":20.013000000000002,"15min":25.613},"min":{"1min":7.5880000000000001,"5min":7.1020000000000003,"15min":6.6799999999999997},"max":{"1min":88.391999999999996,"5min":214.81,"15min":329.49000000000001},"last":19.606000000000002}]},{"osd":5,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":16.939,"5min":18.957999999999998,"15min":24.402000000000001},"min":{"1min":9.1319999999999997,"5min":8.4689999999999994,"15min":7.6449999999999996},"max":{"1min":50.881,"5min":172.79300000000001,"15min":291.26999999999998},"last":12.423},{"interface":"front","average":{"1min":20.372,"5min":22.681000000000001,"15min":29.207000000000001},"min":{"1min":13.125,"5min":11.194000000000001,"15min":11.194000000000001},"max":{"1min":51.552,"5min":173.39699999999999,"15min":310.00700000000001},"last":14.433}]},{"osd":6,"last update":"Mon Oct 30 12:32:54 2023","interfaces":[{"interface":"back","average":{"1min":17.547999999999998,"5min":19.128,"15min":24.890000000000001},"min":{"1min":7.8220000000000001,"5min":7.8220000000000001,"15min":6.7549999999999999},"max":{"1min":98.515000000000001,"5min":160.167,"15min":421.15699999999998},"last":17.768000000000001},{"interface":"front","average":{"1min":20.236000000000001,"5min":22.681999999999999,"15min":26.983000000000001},"min":{"1min":12.522,"5min":9.8710000000000004,"15min":9.8710000000000004},"max":{"1min":53.405999999999999,"5min":143.62899999999999,"15min":315.82299999999998},"last":13.818}]}]},{"osd":1,"up_from":14,"seq":60129542423,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100228,"kb_used_data":6652,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94271612,"statfs":{"total":96636764160,"available":96534130688,"internally_reserved":0,"allocated":6811648,"data_stored":3596255,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":95812038},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:32:59 2023","interfaces":[{"interface":"back","average":{"1min":22.376000000000001,"5min":16.733000000000001,"15min":19.844999999999999},"min":{"1min":6.9429999999999996,"5min":5.1609999999999996,"15min":4.1980000000000004},"max":{"1min":215.155,"5min":215.155,"15min":228.57599999999999},"last":15.202999999999999},{"interface":"front","average":{"1min":21.494,"5min":16.998000000000001,"15min":20.010000000000002},"min":{"1min":5.1710000000000003,"5min":5.1710000000000003,"15min":4.7199999999999998},"max":{"1min":192.125,"5min":192.125,"15min":252.78},"last":12.911}]},{"osd":2,"last update":"Mon Oct 30 12:32:59 2023","interfaces":[{"interface":"back","average":{"1min":34.319000000000003,"5min":24.181999999999999,"15min":27.838999999999999},"min":{"1min":6.5650000000000004,"5min":6.5650000000000004,"15min":6.3339999999999996},"max":{"1min":293.27999999999997,"5min":293.27999999999997,"15min":293.27999999999997},"last":8.0709999999999997},{"interface":"front","average":{"1min":32.225000000000001,"5min":21.768999999999998,"15min":26.533999999999999},"min":{"1min":6.2999999999999998,"5min":6.2000000000000002,"15min":5.8230000000000004},"max":{"1min":294.952,"5min":294.952,"15min":331.928},"last":10.726000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:32:59 2023","interfaces":[{"interface":"back","average":{"1min":24.067,"5min":23.690999999999999,"15min":29.867000000000001},"min":{"1min":7.758,"5min":7.1970000000000001,"15min":6.0229999999999997},"max":{"1min":154.94200000000001,"5min":154.94200000000001,"15min":270.78100000000001},"last":12.129},{"interface":"front","average":{"1min":17.734000000000002,"5min":23.210000000000001,"15min":27.466000000000001},"min":{"1min":9.4390000000000001,"5min":6.633,"15min":6.4960000000000004},"max":{"1min":72.962999999999994,"5min":235.411,"15min":321.42700000000002},"last":16.228000000000002}]},{"osd":4,"last update":"Mon Oct 30 12:32:59 2023","interfaces":[{"interface":"back","average":{"1min":25.596,"5min":24.172999999999998,"15min":27.606999999999999},"min":{"1min":8.9469999999999992,"5min":8.4659999999999993,"15min":6.6890000000000001},"max":{"1min":153.637,"5min":153.637,"15min":278.36399999999998},"last":11.443},{"interface":"front","average":{"1min":20.812999999999999,"5min":22.623999999999999,"15min":25.033000000000001},"min":{"1min":9.7390000000000008,"5min":7.7809999999999997,"15min":6.3280000000000003},"max":{"1min":77.891999999999996,"5min":219.45400000000001,"15min":284.76900000000001},"last":18.812000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:32:46 2023","interfaces":[{"interface":"back","average":{"1min":17.353000000000002,"5min":20.331,"15min":23.489000000000001},"min":{"1min":8.0709999999999997,"5min":7.077,"15min":6.0869999999999997},"max":{"1min":73.831000000000003,"5min":127.604,"15min":281.173},"last":9.9209999999999994},{"interface":"front","average":{"1min":26.169,"5min":26.431999999999999,"15min":26.504000000000001},"min":{"1min":10.946999999999999,"5min":10.473000000000001,"15min":8.2370000000000001},"max":{"1min":141.85499999999999,"5min":280.072,"15min":293.15800000000002},"last":17.114000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:32:59 2023","interfaces":[{"interface":"back","average":{"1min":27.565000000000001,"5min":24.100000000000001,"15min":28.013999999999999},"min":{"1min":10.074999999999999,"5min":6.7089999999999996,"15min":6.7089999999999996},"max":{"1min":157.501,"5min":157.501,"15min":296.68700000000001},"last":9.0980000000000008},{"interface":"front","average":{"1min":21.268000000000001,"5min":22.436,"15min":25.655000000000001},"min":{"1min":9.3889999999999993,"5min":8.2289999999999992,"15min":8.0549999999999997},"max":{"1min":72.337000000000003,"5min":149.857,"15min":288.44099999999997},"last":13.579000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:32:59 2023","interfaces":[{"interface":"back","average":{"1min":36.850000000000001,"5min":26.535,"15min":38.789999999999999},"min":{"1min":11.528,"5min":8.6899999999999995,"15min":8.6899999999999995},"max":{"1min":245.33099999999999,"5min":245.33099999999999,"15min":1115.931},"last":17.984999999999999},{"interface":"front","average":{"1min":40.481999999999999,"5min":30.131,"15min":39.472999999999999},"min":{"1min":12.856,"5min":10.536,"15min":9.5079999999999991},"max":{"1min":246.10900000000001,"5min":246.10900000000001,"15min":1115.2660000000001},"last":19.689}]}]},{"osd":6,"up_from":13,"seq":55834575127,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59996,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311844,"statfs":{"total":96636764160,"available":96575328256,"internally_reserved":0,"allocated":6770688,"data_stored":3198410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:33:00 2023","interfaces":[{"interface":"back","average":{"1min":19.600999999999999,"5min":15.978,"15min":15.167999999999999},"min":{"1min":4.4390000000000001,"5min":3.9380000000000002,"15min":3.9380000000000002},"max":{"1min":110.438,"5min":200.376,"15min":220.25700000000001},"last":10.132999999999999},{"interface":"front","average":{"1min":22.375,"5min":17.699999999999999,"15min":15.675000000000001},"min":{"1min":4.6020000000000003,"5min":4.6020000000000003,"15min":4.1799999999999997},"max":{"1min":145.208,"5min":196.261,"15min":219.40000000000001},"last":11.849}]},{"osd":1,"last update":"Mon Oct 30 12:32:53 2023","interfaces":[{"interface":"back","average":{"1min":21.423999999999999,"5min":19.170000000000002,"15min":18.244},"min":{"1min":8.1899999999999995,"5min":5.1180000000000003,"15min":4.306},"max":{"1min":131.17500000000001,"5min":215.86699999999999,"15min":216.90299999999999},"last":9.423},{"interface":"front","average":{"1min":20.475999999999999,"5min":20.645,"15min":18.667000000000002},"min":{"1min":7.7119999999999997,"5min":5.2039999999999997,"15min":4.5570000000000004},"max":{"1min":143.58799999999999,"5min":225.39699999999999,"15min":225.39699999999999},"last":17.606000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:33:08 2023","interfaces":[{"interface":"back","average":{"1min":14.292999999999999,"5min":18.167999999999999,"15min":18.318999999999999},"min":{"1min":6.0999999999999996,"5min":5.4729999999999999,"15min":5.4729999999999999},"max":{"1min":45.503,"5min":245.411,"15min":245.411},"last":18.209},{"interface":"front","average":{"1min":21.463999999999999,"5min":21.771999999999998,"15min":22.585000000000001},"min":{"1min":9.9320000000000004,"5min":7.0979999999999999,"15min":6.7679999999999998},"max":{"1min":73.278999999999996,"5min":244.76400000000001,"15min":244.76400000000001},"last":11.092000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:33:00 2023","interfaces":[{"interface":"back","average":{"1min":20.994,"5min":23.786999999999999,"15min":20.565999999999999},"min":{"1min":6.7670000000000003,"5min":6.6820000000000004,"15min":5.9649999999999999},"max":{"1min":93.548000000000002,"5min":217.495,"15min":275.60500000000002},"last":16.812000000000001},{"interface":"front","average":{"1min":24.013999999999999,"5min":25.960999999999999,"15min":22.911000000000001},"min":{"1min":10.4,"5min":9.6560000000000006,"15min":8.4960000000000004},"max":{"1min":125.821,"5min":249.18100000000001,"15min":249.18100000000001},"last":19.224}]},{"osd":4,"last update":"Mon Oct 30 12:32:53 2023","interfaces":[{"interface":"back","average":{"1min":21.369,"5min":20.936,"15min":22.457000000000001},"min":{"1min":7.5759999999999996,"5min":7.5759999999999996,"15min":6.556},"max":{"1min":105.456,"5min":246.57599999999999,"15min":246.57599999999999},"last":14.464},{"interface":"front","average":{"1min":14.884,"5min":18.949999999999999,"15min":19.352},"min":{"1min":6.5640000000000001,"5min":6.5640000000000001,"15min":6.5209999999999999},"max":{"1min":52.646999999999998,"5min":233.952,"15min":233.952},"last":6.577}]},{"osd":5,"last update":"Mon Oct 30 12:32:34 2023","interfaces":[{"interface":"back","average":{"1min":29.417999999999999,"5min":24.599,"15min":23.084},"min":{"1min":10,"5min":9.3940000000000001,"15min":7.4080000000000004},"max":{"1min":149.303,"5min":267.37799999999999,"15min":267.37799999999999},"last":15.574},{"interface":"front","average":{"1min":26.631,"5min":21.341000000000001,"15min":21.408999999999999},"min":{"1min":8.3149999999999995,"5min":7.0170000000000003,"15min":5.9089999999999998},"max":{"1min":139.23400000000001,"5min":251.63499999999999,"15min":251.63499999999999},"last":8.8059999999999992}]},{"osd":7,"last update":"Mon Oct 30 12:32:34 2023","interfaces":[{"interface":"back","average":{"1min":31.233000000000001,"5min":25.446000000000002,"15min":23.885999999999999},"min":{"1min":11.182,"5min":9.8510000000000009,"15min":5.7359999999999998},"max":{"1min":142.589,"5min":240.048,"15min":240.048},"last":13.728},{"interface":"front","average":{"1min":24.672000000000001,"5min":24.073,"15min":22.902999999999999},"min":{"1min":9.9990000000000006,"5min":7.7110000000000003,"15min":7.4089999999999998},"max":{"1min":166.40100000000001,"5min":250.70400000000001,"15min":250.70400000000001},"last":20.018000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575127,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78080,"kb_used_data":6648,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293760,"statfs":{"total":96636764160,"available":96556810240,"internally_reserved":0,"allocated":6807552,"data_stored":3596250,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:32:56 2023","interfaces":[{"interface":"back","average":{"1min":17.135999999999999,"5min":16.789000000000001,"15min":14.869},"min":{"1min":4.3869999999999996,"5min":4.0549999999999997,"15min":3.9620000000000002},"max":{"1min":154.18000000000001,"5min":154.18000000000001,"15min":154.18000000000001},"last":5.4169999999999998},{"interface":"front","average":{"1min":15.452,"5min":15.824,"15min":14.702},"min":{"1min":6.625,"5min":5.1280000000000001,"15min":4.3170000000000002},"max":{"1min":85.367000000000004,"5min":128,"15min":128},"last":11.890000000000001}]},{"osd":1,"last update":"Mon Oct 30 12:32:56 2023","interfaces":[{"interface":"back","average":{"1min":19.474,"5min":21.721,"15min":19.373999999999999},"min":{"1min":5.282,"5min":5.282,"15min":5.0410000000000004},"max":{"1min":93.477999999999994,"5min":213.06700000000001,"15min":235.05199999999999},"last":13.26},{"interface":"front","average":{"1min":20.033000000000001,"5min":21.166,"15min":18.876999999999999},"min":{"1min":5.9589999999999996,"5min":5.9589999999999996,"15min":5.2210000000000001},"max":{"1min":84.725999999999999,"5min":187.249,"15min":420.82900000000001},"last":12.608000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:32:56 2023","interfaces":[{"interface":"back","average":{"1min":25.391999999999999,"5min":19.021999999999998,"15min":19.715},"min":{"1min":10.388999999999999,"5min":7.4199999999999999,"15min":5.9539999999999997},"max":{"1min":194.97999999999999,"5min":194.97999999999999,"15min":194.97999999999999},"last":13.920999999999999},{"interface":"front","average":{"1min":21.866,"5min":15.571999999999999,"15min":16.978999999999999},"min":{"1min":6.2640000000000002,"5min":4.8879999999999999,"15min":4.7450000000000001},"max":{"1min":204.173,"5min":204.173,"15min":204.173},"last":7.6749999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:33:01 2023","interfaces":[{"interface":"back","average":{"1min":17.733000000000001,"5min":18.233000000000001,"15min":19.614000000000001},"min":{"1min":7.9249999999999998,"5min":6.2089999999999996,"15min":5.0739999999999998},"max":{"1min":92.182000000000002,"5min":110.496,"15min":165.29400000000001},"last":9.077},{"interface":"front","average":{"1min":23.847999999999999,"5min":21.251000000000001,"15min":22.167999999999999},"min":{"1min":8.4380000000000006,"5min":6.8070000000000004,"15min":6.5330000000000004},"max":{"1min":167.333,"5min":167.333,"15min":167.333},"last":14.694000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:32:56 2023","interfaces":[{"interface":"back","average":{"1min":19.800999999999998,"5min":19.067,"15min":20.035},"min":{"1min":9.1899999999999995,"5min":7.6159999999999997,"15min":6.1299999999999999},"max":{"1min":115.056,"5min":115.056,"15min":265.50400000000002},"last":15.484999999999999},{"interface":"front","average":{"1min":23.84,"5min":20.538,"15min":19.704000000000001},"min":{"1min":10.19,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":163.90100000000001,"5min":163.90100000000001,"15min":163.90100000000001},"last":16.106999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:32:56 2023","interfaces":[{"interface":"back","average":{"1min":22.785,"5min":21.588000000000001,"15min":19.966999999999999},"min":{"1min":11.532999999999999,"5min":8.1229999999999993,"15min":8.1229999999999993},"max":{"1min":114.43899999999999,"5min":114.43899999999999,"15min":150.97200000000001},"last":16.785},{"interface":"front","average":{"1min":25.847000000000001,"5min":23.702999999999999,"15min":21.257000000000001},"min":{"1min":9.9510000000000005,"5min":7.9870000000000001,"15min":7.3460000000000001},"max":{"1min":161.34800000000001,"5min":182.376,"15min":182.376},"last":18.728999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:32:56 2023","interfaces":[{"interface":"back","average":{"1min":19.847999999999999,"5min":22.814,"15min":24.838000000000001},"min":{"1min":7.242,"5min":7.242,"15min":6.96},"max":{"1min":123.27200000000001,"5min":124.621,"15min":854.64300000000003},"last":10.099},{"interface":"front","average":{"1min":25.837,"5min":18.731000000000002,"15min":17.510000000000002},"min":{"1min":10.747999999999999,"5min":6.2670000000000003,"15min":5.5300000000000002},"max":{"1min":207.78,"5min":207.78,"15min":264.84699999999998},"last":10.775}]}]},{"osd":4,"up_from":13,"seq":55834575127,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82204,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289636,"statfs":{"total":96636764160,"available":96552587264,"internally_reserved":0,"allocated":6770688,"data_stored":3198410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:32:57 2023","interfaces":[{"interface":"back","average":{"1min":18.170999999999999,"5min":15.954000000000001,"15min":19.619},"min":{"1min":5.6289999999999996,"5min":5.1920000000000002,"15min":4.617},"max":{"1min":118.92400000000001,"5min":139.40199999999999,"15min":372.52300000000002},"last":10.981},{"interface":"front","average":{"1min":23.437000000000001,"5min":20.744,"15min":23.27},"min":{"1min":8.8390000000000004,"5min":6.641,"15min":6.0110000000000001},"max":{"1min":144.375,"5min":144.375,"15min":200.37299999999999},"last":14.781000000000001}]},{"osd":1,"last update":"Mon Oct 30 12:32:32 2023","interfaces":[{"interface":"back","average":{"1min":24.058,"5min":20.231000000000002,"15min":22.433},"min":{"1min":5.5039999999999996,"5min":5.5039999999999996,"15min":5.5039999999999996},"max":{"1min":126.119,"5min":167.84700000000001,"15min":231.49799999999999},"last":13.907},{"interface":"front","average":{"1min":20.042999999999999,"5min":16.888000000000002,"15min":19.818999999999999},"min":{"1min":5.79,"5min":5.2690000000000001,"15min":4.641},"max":{"1min":112.584,"5min":223.441,"15min":223.441},"last":10.112}]},{"osd":2,"last update":"Mon Oct 30 12:32:32 2023","interfaces":[{"interface":"back","average":{"1min":15.933999999999999,"5min":23.056000000000001,"15min":26.792000000000002},"min":{"1min":6.2549999999999999,"5min":6.2549999999999999,"15min":6.2149999999999999},"max":{"1min":55.453000000000003,"5min":198.66999999999999,"15min":438.15499999999997},"last":15.523999999999999},{"interface":"front","average":{"1min":14.124000000000001,"5min":18.303000000000001,"15min":21.728000000000002},"min":{"1min":6.5380000000000003,"5min":6.2220000000000004,"15min":4.944},"max":{"1min":41.518999999999998,"5min":254.06399999999999,"15min":254.06399999999999},"last":11.941000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:33:09 2023","interfaces":[{"interface":"back","average":{"1min":16.696000000000002,"5min":21.370999999999999,"15min":27.033999999999999},"min":{"1min":6.8890000000000002,"5min":6.8890000000000002,"15min":6.8890000000000002},"max":{"1min":47.145000000000003,"5min":123.752,"15min":269.06700000000001},"last":8.8970000000000002},{"interface":"front","average":{"1min":15.683999999999999,"5min":15.401999999999999,"15min":20.497},"min":{"1min":5.2400000000000002,"5min":4.8230000000000004,"15min":4.8230000000000004},"max":{"1min":100.45,"5min":100.45,"15min":239.715},"last":17.556999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:32:57 2023","interfaces":[{"interface":"back","average":{"1min":24.782,"5min":22.007000000000001,"15min":23.882999999999999},"min":{"1min":7.5679999999999996,"5min":7.5679999999999996,"15min":7.5679999999999996},"max":{"1min":147.47900000000001,"5min":147.47900000000001,"15min":268.34199999999998},"last":16.844000000000001},{"interface":"front","average":{"1min":24.309000000000001,"5min":18.463000000000001,"15min":20.356999999999999},"min":{"1min":8.2379999999999995,"5min":6.4000000000000004,"15min":5.452},"max":{"1min":160.215,"5min":160.215,"15min":252.899},"last":12.625999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:32:57 2023","interfaces":[{"interface":"back","average":{"1min":24.315999999999999,"5min":19.625,"15min":24.132999999999999},"min":{"1min":10.522,"5min":8.8559999999999999,"15min":7.21},"max":{"1min":91.209999999999994,"5min":93.864000000000004,"15min":273.90100000000001},"last":20.091999999999999},{"interface":"front","average":{"1min":14.233000000000001,"5min":16.449000000000002,"15min":26.591999999999999},"min":{"1min":6.5090000000000003,"5min":5.8849999999999998,"15min":5.8849999999999998},"max":{"1min":46.805999999999997,"5min":80.379000000000005,"15min":292.11099999999999},"last":6.9909999999999997}]},{"osd":7,"last update":"Mon Oct 30 12:32:57 2023","interfaces":[{"interface":"back","average":{"1min":36.566000000000003,"5min":26.021000000000001,"15min":30.43},"min":{"1min":10.157,"5min":9.0250000000000004,"15min":6.46},"max":{"1min":193.06899999999999,"5min":193.06899999999999,"15min":652.50699999999995},"last":18.260999999999999},{"interface":"front","average":{"1min":34.810000000000002,"5min":24.241,"15min":27.437999999999999},"min":{"1min":14.51,"5min":10.422000000000001,"15min":8.9309999999999992},"max":{"1min":183.166,"5min":183.166,"15min":303.19999999999999},"last":21.169}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":2248,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:33:21.445 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-30T12:33:21.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.550+0000 7f7278bbb640 10 monclient: get_monmap_and_config 2023-10-30T12:33:21.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.550+0000 7f7278bbb640 10 monclient: build_initial_monmap 2023-10-30T12:33:21.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.550+0000 7f7278bbb640 10 monclient: monmap: 2023-10-30T12:33:21.552 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:33:21.552 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:33:21.552 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:33:21.551212+0000 2023-10-30T12:33:21.553 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:33:21.551212+0000 2023-10-30T12:33:21.553 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:33:21.553 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:33:21.553 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a-legacy 2023-10-30T12:33:21.553 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-10-30T12:33:21.554 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-b 2023-10-30T12:33:21.554 INFO:teuthology.orchestra.run.smithi003.stderr:3: v2:172.21.15.3:3300/0 mon.noname-a 2023-10-30T12:33:21.554 INFO:teuthology.orchestra.run.smithi003.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-10-30T12:33:21.554 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:21.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.550+0000 7f7278bbb640 1 Processor -- start 2023-10-30T12:33:21.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.550+0000 7f7278bbb640 1 -- start start 2023-10-30T12:33:21.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.550+0000 7f7278bbb640 10 monclient: init 2023-10-30T12:33:21.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.551+0000 7f7278bbb640 10 monclient: _reopen_session rank -1 2023-10-30T12:33:21.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.551+0000 7f7278bbb640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-10-30T12:33:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.551+0000 7f7278bbb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f72740b4350 0x7f72740a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.551+0000 7f7278bbb640 10 monclient(hunting): picked mon.noname-b con 0x7f72740b4350 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:33:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.551+0000 7f7278bbb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f727413eb50 addr v1:172.21.15.92:6789/0 2023-10-30T12:33:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.551+0000 7f7278bbb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7274144f50 addr v1:172.21.15.3:6789/0 2023-10-30T12:33:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.551+0000 7f7278bbb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72740aa6e0 con 0x7f7274144f50 2023-10-30T12:33:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.551+0000 7f7278bbb640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f727413f680 con 0x7f727413eb50 2023-10-30T12:33:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.551+0000 7f7278bbb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:33:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.551+0000 7f7278bbb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72740679f0 con 0x7f72740b4350 2023-10-30T12:33:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.551+0000 7f7278bbb640 10 monclient(hunting): _renew_subs 2023-10-30T12:33:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.551+0000 7f7278bbb640 10 monclient(hunting): authenticate will time out at 2792.765691s 2023-10-30T12:33:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.552+0000 7f7272d9d640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7274144f50 0x7f72741453e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:40478/0 (socket says 172.21.15.3:40478) 2023-10-30T12:33:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.552+0000 7f7272d9d640 1 -- 172.21.15.3:0/524350117 learned_addr learned my addr 172.21.15.3:0/524350117 (peer_addr_for_me v1:172.21.15.3:0/0) 2023-10-30T12:33:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.552+0000 7f727259c640 1 --2- 172.21.15.3:0/524350117 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f72740b4350 0x7f72740a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.552+0000 7f727259c640 10 monclient(hunting): get_auth_request con 0x7f72740b4350 auth_method 0 2023-10-30T12:33:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.552+0000 7f727259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:33:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.552+0000 7f727259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:33:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.552+0000 7f727259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:33:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.553+0000 7f727259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:33:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.553+0000 7f727259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:33:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.553+0000 7f727259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:33:21.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727159a640 1 -- 172.21.15.3:0/524350117 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522058841 0 0) 0x7f725c001110 con 0x7f727413eb50 2023-10-30T12:33:21.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727159a640 10 monclient(hunting): _negotiate 2023-10-30T12:33:21.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727159a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:33:21.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727159a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:33:21.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727159a640 10 monclient(hunting): my global_id is 21916 2023-10-30T12:33:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727159a640 1 -- 172.21.15.3:0/524350117 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7260003520 con 0x7f727413eb50 2023-10-30T12:33:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727159a640 1 -- 172.21.15.3:0/524350117 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418151272 0 0) 0x7f726c0013c0 con 0x7f7274144f50 2023-10-30T12:33:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727159a640 10 monclient(hunting): _negotiate 2023-10-30T12:33:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727159a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:33:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727159a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:33:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727159a640 10 monclient(hunting): my global_id is 16710 2023-10-30T12:33:21.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727159a640 1 -- 172.21.15.3:0/524350117 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7260003840 con 0x7f7274144f50 2023-10-30T12:33:21.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727259c640 10 monclient(hunting): handle_auth_done global_id 15629 payload 274 2023-10-30T12:33:21.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727259c640 1 -- 172.21.15.3:0/524350117 >> v1:172.21.15.92:6789/0 conn(0x7f727413eb50 legacy=0x7f727413ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727259c640 1 -- 172.21.15.3:0/524350117 >> v1:172.21.15.3:6789/0 conn(0x7f7274144f50 legacy=0x7f72741453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727259c640 10 monclient: _finish_hunting 0 2023-10-30T12:33:21.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727259c640 1 monclient: found mon.noname-b 2023-10-30T12:33:21.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:33:21.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.3:3301/0 2023-10-30T12:33:21.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.554+0000 7f727259c640 1 -- 172.21.15.3:0/524350117 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72740a8910 con 0x7f72740b4350 2023-10-30T12:33:21.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.557+0000 7f727259c640 1 --2- 172.21.15.3:0/524350117 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f72740b4350 0x7f72740a83d0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7f72640097d0 tx=0x7f726402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66d310223a45c18b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f727159a640 1 -- 172.21.15.3:0/524350117 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7264031770 con 0x7f72740b4350 2023-10-30T12:33:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f727159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:33:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f727159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:33:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f727159a640 10 monclient: dump: 2023-10-30T12:33:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:33:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:33:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:33:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:33:21.572 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:33:21.572 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:33:21.572 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:33:21.572 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:21.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f727159a640 10 monclient: _finish_auth 0 2023-10-30T12:33:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f727159a640 10 monclient: _check_auth_tickets 2023-10-30T12:33:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f727159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:33:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f727159a640 1 -- 172.21.15.3:0/524350117 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72640318e0 con 0x7f72740b4350 2023-10-30T12:33:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f7278bbb640 5 monclient: authenticate success, global_id 15629 2023-10-30T12:33:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f7278bbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:33:21.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f727159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:33:21.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f7278bbb640 10 monclient: get_monmap_and_config success 2023-10-30T12:33:21.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f7278bbb640 4 set_mon_vals no callback set 2023-10-30T12:33:21.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f727159a640 1 -- 172.21.15.3:0/524350117 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7264031c80 con 0x7f72740b4350 2023-10-30T12:33:21.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f727159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:33:21.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f727159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:33:21.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f727159a640 10 monclient: dump: 2023-10-30T12:33:21.575 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:33:21.575 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:33:21.575 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:33:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:33:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:33:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:33:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:33:21.577 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f7278bbb640 10 monclient: shutdown 2023-10-30T12:33:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f7278bbb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:33:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f7278bbb640 1 -- 172.21.15.3:0/524350117 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f72740b4350 msgr2=0x7f72740a83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f7278bbb640 1 --2- 172.21.15.3:0/524350117 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f72740b4350 0x7f72740a83d0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7f72640097d0 tx=0x7f726402b810 comp rx=0 tx=0).stop 2023-10-30T12:33:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f7278bbb640 1 -- 172.21.15.3:0/524350117 shutdown_connections 2023-10-30T12:33:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f7278bbb640 1 --2- 172.21.15.3:0/524350117 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f72740b4350 0x7f72740a83d0 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f7278bbb640 1 -- 172.21.15.3:0/524350117 >> 172.21.15.3:0/524350117 conn(0x7f7274065ea0 msgr2=0x7f72740662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f7278bbb640 1 -- 172.21.15.3:0/524350117 shutdown_connections 2023-10-30T12:33:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.561+0000 7f7278bbb640 1 -- 172.21.15.3:0/524350117 wait complete. 2023-10-30T12:33:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.562+0000 7f7278bbb640 10 monclient: build_initial_monmap 2023-10-30T12:33:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.562+0000 7f7278bbb640 10 monclient: monmap: 2023-10-30T12:33:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2023-10-30T12:33:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:33:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2023-10-30T12:33:21.580 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2023-10-30T12:33:21.580 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2023-10-30T12:33:21.580 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2023-10-30T12:33:21.580 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.noname-a 2023-10-30T12:33:21.580 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-10-30T12:33:21.581 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.noname-c 2023-10-30T12:33:21.581 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:21.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.563+0000 7f7278bbb640 1 librados: starting msgr at 2023-10-30T12:33:21.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.563+0000 7f7278bbb640 1 librados: starting objecter 2023-10-30T12:33:21.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.563+0000 7f7278bbb640 1 Processor -- start 2023-10-30T12:33:21.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.563+0000 7f7278bbb640 1 -- start start 2023-10-30T12:33:21.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.563+0000 7f7278bbb640 1 librados: setting wanted keys 2023-10-30T12:33:21.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.563+0000 7f7278bbb640 1 librados: calling monclient init 2023-10-30T12:33:21.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.563+0000 7f7278bbb640 10 monclient: init 2023-10-30T12:33:21.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.563+0000 7f7278bbb640 10 monclient: _reopen_session rank -1 2023-10-30T12:33:21.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.563+0000 7f7278bbb640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:33:21.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7278bbb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72740b4350 0x7f72741da4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7278bbb640 10 monclient(hunting): picked mon.noname-a con 0x7f72740b4350 addr [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] 2023-10-30T12:33:21.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7278bbb640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f727413eb50 0x7f72741da9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7278bbb640 10 monclient(hunting): picked mon.noname-b con 0x7f727413eb50 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-10-30T12:33:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7278bbb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7274144f50 0x7f72741d6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7278bbb640 10 monclient(hunting): picked mon.noname-c con 0x7f7274144f50 addr [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2023-10-30T12:33:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7278bbb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:33:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7278bbb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727413f020 con 0x7f72740b4350 2023-10-30T12:33:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7278bbb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:33:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7278bbb640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72741da050 con 0x7f727413eb50 2023-10-30T12:33:21.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7278bbb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:33:21.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7278bbb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7274141b50 con 0x7f7274144f50 2023-10-30T12:33:21.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7278bbb640 10 monclient(hunting): _renew_subs 2023-10-30T12:33:21.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7278bbb640 10 monclient(hunting): authenticate will time out at 2792.778129s 2023-10-30T12:33:21.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f727259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72740b4350 0x7f72741da4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7272d9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7274144f50 0x7f72741d6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7271d9b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f727413eb50 0x7f72741da9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f727259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72740b4350 0x7f72741da4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34730/0 (socket says 172.21.15.3:34730) 2023-10-30T12:33:21.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f727259c640 1 -- 172.21.15.3:0/1165514863 learned_addr learned my addr 172.21.15.3:0/1165514863 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-10-30T12:33:21.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f727259c640 10 monclient(hunting): get_auth_request con 0x7f72740b4350 auth_method 0 2023-10-30T12:33:21.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f727259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:33:21.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f727259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:33:21.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f727259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:33:21.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7272d9d640 10 monclient(hunting): get_auth_request con 0x7f7274144f50 auth_method 0 2023-10-30T12:33:21.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7272d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:33:21.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7272d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:33:21.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7272d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:33:21.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7271d9b640 10 monclient(hunting): get_auth_request con 0x7f727413eb50 auth_method 0 2023-10-30T12:33:21.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7271d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:33:21.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7271d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:33:21.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.564+0000 7f7271d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:33:21.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.565+0000 7f727259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:33:21.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.565+0000 7f727259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:33:21.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.565+0000 7f727259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:33:21.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.565+0000 7f7271d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:33:21.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.565+0000 7f7271d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:33:21.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.565+0000 7f7271d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:33:21.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.565+0000 7f7272d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:33:21.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.565+0000 7f7272d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:33:21.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.565+0000 7f7272d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:33:21.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.567+0000 7f7271d9b640 10 monclient(hunting): handle_auth_done global_id 21919 payload 824 2023-10-30T12:33:21.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.567+0000 7f7271d9b640 1 -- 172.21.15.3:0/1165514863 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7274144f50 msgr2=0x7f72741d6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.567+0000 7f7271d9b640 1 --2- 172.21.15.3:0/1165514863 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7274144f50 0x7f72741d6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.567+0000 7f7271d9b640 1 -- 172.21.15.3:0/1165514863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72740b4350 msgr2=0x7f72741da4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.567+0000 7f7271d9b640 1 --2- 172.21.15.3:0/1165514863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72740b4350 0x7f72741da4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.567+0000 7f7271d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:33:21.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.567+0000 7f7271d9b640 1 monclient: found mon.noname-b 2023-10-30T12:33:21.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.567+0000 7f7271d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:33:21.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.567+0000 7f7271d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-10-30T12:33:21.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.567+0000 7f7271d9b640 1 -- 172.21.15.3:0/1165514863 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72741d69a0 con 0x7f727413eb50 2023-10-30T12:33:21.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.570+0000 7f7271d9b640 1 --2- 172.21.15.3:0/1165514863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f727413eb50 0x7f72741da9e0 secure :-1 s=READY pgs=5331 cs=0 l=1 rev1=1 crypto rx=0x7f725c00c3f0 tx=0x7f725c00c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:21.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.570+0000 7f726affd640 10 client.?.objecter ms_handle_connect 0x7f727413eb50 2023-10-30T12:33:21.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.570+0000 7f726affd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:33:21.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f726affd640 1 -- 172.21.15.3:0/1165514863 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c0052d0 con 0x7f727413eb50 2023-10-30T12:33:21.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f726affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:33:21.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f726affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:33:21.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f726affd640 10 monclient: dump: 2023-10-30T12:33:21.595 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:33:21.595 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:33:21.595 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:21.596 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:21.596 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:33:21.596 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:33:21.596 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:33:21.596 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:33:21.596 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:33:21.597 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:21.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f726affd640 10 monclient: _finish_auth 0 2023-10-30T12:33:21.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f726affd640 10 monclient: _check_auth_tickets 2023-10-30T12:33:21.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f726affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:33:21.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f726affd640 1 -- 172.21.15.3:0/1165514863 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725c00e040 con 0x7f727413eb50 2023-10-30T12:33:21.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f726affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:33:21.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f726affd640 1 -- 172.21.15.3:0/1165514863 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c0035f0 con 0x7f727413eb50 2023-10-30T12:33:21.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f726affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:33:21.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f726affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:33:21.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f726affd640 10 monclient: dump: 2023-10-30T12:33:21.599 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2023-10-30T12:33:21.599 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 9bae44ae-7379-4ab2-a454-4d550607110c 2023-10-30T12:33:21.599 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:21.599 INFO:teuthology.orchestra.run.smithi003.stderr:created 2023-10-30T12:08:26.869942+0000 2023-10-30T12:33:21.599 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 18 (reef) 2023-10-30T12:33:21.600 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2023-10-30T12:33:21.600 INFO:teuthology.orchestra.run.smithi003.stderr:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-30T12:33:21.600 INFO:teuthology.orchestra.run.smithi003.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-10-30T12:33:21.600 INFO:teuthology.orchestra.run.smithi003.stderr:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2023-10-30T12:33:21.600 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:33:21.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f7278bbb640 5 monclient: authenticate success, global_id 21919 2023-10-30T12:33:21.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f7278bbb640 10 monclient: _renew_subs 2023-10-30T12:33:21.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f7278bbb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:33:21.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f7278bbb640 1 -- 172.21.15.3:0/1165514863 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72741d7440 con 0x7f727413eb50 2023-10-30T12:33:21.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f7270d99640 4 set_mon_vals no callback set 2023-10-30T12:33:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f7278bbb640 10 client.21919.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:33:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f7278bbb640 10 monclient: _renew_subs 2023-10-30T12:33:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f7278bbb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:33:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f7278bbb640 1 -- 172.21.15.3:0/1165514863 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72741d7910 con 0x7f727413eb50 2023-10-30T12:33:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.573+0000 7f7278bbb640 1 librados: init done 2023-10-30T12:33:21.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.574+0000 7f7278bbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:33:21.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.574+0000 7f7278bbb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:33:21.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.574+0000 7f7278bbb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:33:21.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.574+0000 7f7278bbb640 1 -- 172.21.15.3:0/1165514863 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f727404e9b0 con 0x7f727413eb50 2023-10-30T12:33:21.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.815+0000 7f726affd640 1 -- 172.21.15.3:0/1165514863 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f725c003760 con 0x7f727413eb50 2023-10-30T12:33:21.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.815+0000 7f726affd640 1 --2- 172.21.15.3:0/1165514863 >> v2:172.21.15.92:6800/1750155735 conn(0x7f725406c810 0x7f725406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.815+0000 7f727259c640 1 --2- 172.21.15.3:0/1165514863 >> v2:172.21.15.92:6800/1750155735 conn(0x7f725406c810 0x7f725406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.815+0000 7f727259c640 10 monclient: get_auth_request con 0x7f725406c810 auth_method 0 2023-10-30T12:33:21.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.816+0000 7f726affd640 10 client.21919.objecter ms_handle_connect 0x7f725406c810 2023-10-30T12:33:21.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.816+0000 7f727259c640 1 --2- 172.21.15.3:0/1165514863 >> v2:172.21.15.92:6800/1750155735 conn(0x7f725406c810 0x7f725406ecd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7264000c00 tx=0x7f726403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:21.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.817+0000 7f726affd640 1 -- 172.21.15.3:0/1165514863 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(900..900 src has 627..900) v4 ==== 5081+0+0 (secure 0 0 0) 0x7f725c057770 con 0x7f727413eb50 2023-10-30T12:33:21.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.817+0000 7f726affd640 10 client.21919.objecter ms_dispatch 0x7f7274000b90 osd_map(900..900 src has 627..900) v4 2023-10-30T12:33:21.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.817+0000 7f726affd640 3 client.21919.objecter handle_osd_map got epochs [900,900] > 0 2023-10-30T12:33:21.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.817+0000 7f726affd640 3 client.21919.objecter handle_osd_map decoding full epoch 900 2023-10-30T12:33:21.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.817+0000 7f726affd640 20 client.21919.objecter dump_active .. 0 homeless 2023-10-30T12:33:21.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.827+0000 7f726affd640 1 -- 172.21.15.3:0/1165514863 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f725c05b810 con 0x7f727413eb50 2023-10-30T12:33:21.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.827+0000 7f726affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:33:21.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:21.827+0000 7f726affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:33:22.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.005+0000 7f7278bbb640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-10-30T12:33:22.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.005+0000 7f7278bbb640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-10-30T12:33:22.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.005+0000 7f7278bbb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-10-30T12:33:22.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.005+0000 7f7278bbb640 1 -- 172.21.15.3:0/1165514863 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f72741d69a0 con 0x7f727413eb50 2023-10-30T12:33:22.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.248+0000 7f726affd640 1 -- 172.21.15.3:0/1165514863 <== mon.1 v2:172.21.15.92:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f725c086e80 con 0x7f727413eb50 2023-10-30T12:33:22.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.248+0000 7f726affd640 10 monclient: handle_config config(1 keys) v1 2023-10-30T12:33:22.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.248+0000 7f7270d99640 4 set_mon_vals no callback set 2023-10-30T12:33:22.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.248+0000 7f7270d99640 10 set_mon_vals mon_health_to_clog = false 2023-10-30T12:33:22.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.248+0000 7f726affd640 1 -- 172.21.15.3:0/1165514863 <== mon.1 v2:172.21.15.92:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7f725c05f9c0 con 0x7f727413eb50 2023-10-30T12:33:22.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.248+0000 7f726affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-10-30T12:33:22.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.248+0000 7f726affd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:33:22.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 10 librados: watch_flush enter 2023-10-30T12:33:22.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 10 librados: watch_flush exit 2023-10-30T12:33:22.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 20 client.21919.objecter shutdown clearing up homeless session... 2023-10-30T12:33:22.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 10 client.21919.objecter successfully canceled tick 2023-10-30T12:33:22.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 1 -- 172.21.15.3:0/1165514863 >> v2:172.21.15.92:6800/1750155735 conn(0x7f725406c810 msgr2=0x7f725406ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:22.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 1 --2- 172.21.15.3:0/1165514863 >> v2:172.21.15.92:6800/1750155735 conn(0x7f725406c810 0x7f725406ecd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7264000c00 tx=0x7f726403d040 comp rx=0 tx=0).stop 2023-10-30T12:33:22.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 10 monclient: shutdown 2023-10-30T12:33:22.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:33:22.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 1 -- 172.21.15.3:0/1165514863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f727413eb50 msgr2=0x7f72741da9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:22.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 1 --2- 172.21.15.3:0/1165514863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f727413eb50 0x7f72741da9e0 secure :-1 s=READY pgs=5331 cs=0 l=1 rev1=1 crypto rx=0x7f725c00c3f0 tx=0x7f725c00c8b0 comp rx=0 tx=0).stop 2023-10-30T12:33:22.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 1 -- 172.21.15.3:0/1165514863 shutdown_connections 2023-10-30T12:33:22.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 1 --2- 172.21.15.3:0/1165514863 >> v2:172.21.15.92:6800/1750155735 conn(0x7f725406c810 0x7f725406ecd0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:22.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 1 --2- 172.21.15.3:0/1165514863 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7274144f50 0x7f72741d6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:22.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 1 --2- 172.21.15.3:0/1165514863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f727413eb50 0x7f72741da9e0 unknown :-1 s=CLOSED pgs=5331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:22.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 1 --2- 172.21.15.3:0/1165514863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72740b4350 0x7f72741da4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:22.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 1 -- 172.21.15.3:0/1165514863 >> 172.21.15.3:0/1165514863 conn(0x7f7274065ea0 msgr2=0x7f72741430d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:22.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 1 -- 172.21.15.3:0/1165514863 shutdown_connections 2023-10-30T12:33:22.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 1 -- 172.21.15.3:0/1165514863 wait complete. 2023-10-30T12:33:22.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-30T12:33:22.250+0000 7f7278bbb640 1 librados: shutdown 2023-10-30T12:33:22.263 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-30T12:33:22.263 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-30T12:33:22.263 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-30T12:33:22.264 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:33:22.278 INFO:tasks.ceph.osd.0.smithi003.stderr:2023-10-30T12:33:22.266+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86722) UID: 0 2023-10-30T12:33:22.279 INFO:tasks.ceph.osd.0.smithi003.stderr:2023-10-30T12:33:22.267+0000 ac7d640 -1 osd.0 900 *** Got signal Terminated *** 2023-10-30T12:33:27.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-30T12:33:28.366 INFO:tasks.ceph.osd.0:Stopped 2023-10-30T12:33:28.366 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-30T12:33:28.367 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:33:28.380 INFO:tasks.ceph.osd.1.smithi003.stderr:2023-10-30T12:33:28.369+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86727) UID: 0 2023-10-30T12:33:28.380 INFO:tasks.ceph.osd.1.smithi003.stderr:2023-10-30T12:33:28.371+0000 ac7d640 -1 osd.1 902 *** Got signal Terminated *** 2023-10-30T12:33:33.648 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-30T12:33:38.206 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:38.142 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:38.208 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:38.144 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:38.230 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:38.165 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:38.260 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:38.195 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:38.261 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:38.197 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:38.264 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:38.173 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:38.266 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:38.174 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:38.279 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:38.214 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:38.298 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:38.207 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:38.953 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:38.862 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:38.954 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:38.863 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:38.962 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:38.871 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:38.995 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:38.930 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:38.996 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:38.932 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:39.005 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:38.940 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:39.052 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:38.987 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:39.053 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:38.989 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:39.062 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:38.997 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:40.739 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:40.674 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:40.750 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:40.658 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:40.751 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:40.687 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:40.788 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:40.723 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:40.796 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:40.705 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:40.797 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:40.732 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:40.801 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:40.736 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:40.810 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:40.719 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:40.812 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:40.748 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:58.196 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:58.131 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:58.246 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:58.181 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:58.247 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:58.182 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:58.249 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:58.184 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:58.267 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:58.202 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:58.268 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:58.204 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:58.296 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:58.204 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:58.299 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:58.208 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:58.327 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:58.236 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:58.928 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:58.863 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:58.929 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:58.865 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:58.938 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:24:58.873 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:58.939 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:58.848 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:58.940 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:58.849 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:58.948 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:58.857 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:59.010 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:58.945 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:59.011 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:58.946 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:33:59.019 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:24:58.955 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:18.297 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:25:18.233 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:18.300 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:25:18.236 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:18.315 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:25:18.250 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:18.345 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:18.254 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:18.348 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:18.256 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:18.349 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:25:18.284 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:18.350 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:25:18.286 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:18.363 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:18.271 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:18.366 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:25:18.302 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:18.993 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:18.901 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:18.994 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:18.902 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:19.005 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:18.912 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:19.056 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:25:18.992 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:19.057 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:25:18.993 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:19.070 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:25:19.005 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:19.071 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:25:19.006 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:19.072 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:25:19.008 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:19.081 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:25:19.017 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:33.374 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:25:33.309 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:33.376 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:25:33.311 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:33.396 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:25:33.331 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:33.425 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:33.333 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:33.426 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:33.335 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:33.441 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:33.350 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:33.443 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:25:33.379 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:33.447 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:25:33.383 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:33.465 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:25:33.400 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.404 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:25:38.339 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.406 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:25:38.341 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.416 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:25:38.351 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.447 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:25:38.382 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.448 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:38.357 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.451 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:38.360 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.452 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:25:38.387 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.461 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:38.370 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:38.465 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:25:38.400 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:58.452 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:25:58.388 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:58.453 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:25:58.389 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:58.475 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:25:58.410 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:58.496 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:58.405 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:58.499 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:58.408 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:58.499 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:25:58.434 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:58.502 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:25:58.437 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:58.514 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:58.423 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:34:58.524 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:25:58.460 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:03.468 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:03.403 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:03.470 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:03.405 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:03.486 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:03.422 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:03.515 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:03.424 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:03.517 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:03.426 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:03.527 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:03.436 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:03.541 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:03.476 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:03.544 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:03.480 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:03.558 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:03.493 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:18.510 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:18.445 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:18.511 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:18.447 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:18.531 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:18.467 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:18.560 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:18.468 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:18.563 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:18.472 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:18.564 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:18.500 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:18.566 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:18.501 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:18.581 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:18.516 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:18.582 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:18.491 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:23.518 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:23.453 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:23.522 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:23.457 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:23.534 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:23.469 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:23.565 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:23.474 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:23.568 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:23.503 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:23.570 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:23.478 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:23.571 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:23.507 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:23.580 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:23.489 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:23.585 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:23.520 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:38.563 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:38.498 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:38.564 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:38.499 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:38.577 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:38.512 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:38.610 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:38.545 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:38.612 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:38.547 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:38.612 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:38.521 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:38.614 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:38.522 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:38.627 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:38.536 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:38.630 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:38.565 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:43.573 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:43.508 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:43.575 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:43.511 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:43.589 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:43.524 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:43.624 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:43.532 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:43.624 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:43.560 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:43.625 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:43.534 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:43.627 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:43.562 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:43.636 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:43.544 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:43.641 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:43.576 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:58.616 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:58.552 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:58.618 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:58.553 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:58.631 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:26:58.567 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:58.667 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:58.575 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:58.667 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:58.602 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:58.668 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:58.576 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:58.669 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:58.604 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:58.681 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:58.590 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:35:58.684 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:26:58.619 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:03.626 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:03.561 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:03.629 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:03.565 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:03.641 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:03.577 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:03.678 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:03.586 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:03.679 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:03.588 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:03.688 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:03.597 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:03.695 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:03.630 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:03.696 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:03.632 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:03.707 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:03.642 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:18.672 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:18.607 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:18.674 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:18.609 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:18.688 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:18.623 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:18.718 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:18.627 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:18.720 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:18.629 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:18.721 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:18.657 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:18.724 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:18.659 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:18.735 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:18.644 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:18.740 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:18.675 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:19.210 INFO:tasks.ceph.osd.1:Stopped 2023-10-30T12:36:19.211 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-30T12:36:19.211 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:36:19.224 INFO:tasks.ceph.osd.2.smithi003.stderr:2023-10-30T12:36:19.213+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86738) UID: 0 2023-10-30T12:36:19.224 INFO:tasks.ceph.osd.2.smithi003.stderr:2023-10-30T12:36:19.215+0000 ac7d640 -1 osd.2 904 *** Got signal Terminated *** 2023-10-30T12:36:23.683 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:23.618 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:23.685 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:23.621 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:23.699 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:23.635 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:23.729 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:23.638 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:23.732 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:23.640 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:23.743 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:23.651 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:23.754 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:23.689 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:23.756 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:23.692 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:23.768 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:23.704 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:25.312 INFO:tasks.ceph.osd.2:Stopped 2023-10-30T12:36:25.313 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-30T12:36:25.313 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:36:25.326 INFO:tasks.ceph.osd.3.smithi003.stderr:2023-10-30T12:36:25.316+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86739) UID: 0 2023-10-30T12:36:25.329 INFO:tasks.ceph.osd.3.smithi003.stderr:2023-10-30T12:36:25.317+0000 ac7d640 -1 osd.3 906 *** Got signal Terminated *** 2023-10-30T12:36:31.416 INFO:tasks.ceph.osd.3:Stopped 2023-10-30T12:36:31.416 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-30T12:36:31.417 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:36:31.429 INFO:tasks.ceph.osd.4.smithi092.stderr:2023-10-30T12:36:31.418+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86360) UID: 0 2023-10-30T12:36:31.429 INFO:tasks.ceph.osd.4.smithi092.stderr:2023-10-30T12:36:31.420+0000 ac7d640 -1 osd.4 908 *** Got signal Terminated *** 2023-10-30T12:36:33.746 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:33.681 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:33.748 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:33.684 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:33.763 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:33.699 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:33.793 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:33.729 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:33.796 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:33.731 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:33.813 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:33.722 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:33.816 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:33.724 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:33.819 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:33.754 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:33.831 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:33.740 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:43.742 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:43.677 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:43.747 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:43.682 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:43.767 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:43.702 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:43.793 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:43.702 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:43.794 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:43.703 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:43.801 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:43.736 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:43.804 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:43.740 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:43.807 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:43.716 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:43.817 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:43.752 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:48.468 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:48.403 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:48.471 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:48.406 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:48.487 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:48.422 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:48.516 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:48.424 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:48.520 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:48.429 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:48.526 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:48.461 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:48.527 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:48.463 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:48.533 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:48.442 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:48.542 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:48.478 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:53.797 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:53.730 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:53.799 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:53.735 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:53.811 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:27:53.746 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:53.843 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:53.752 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:53.847 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:53.755 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:53.848 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:53.783 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:53.851 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:53.786 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:53.857 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:53.765 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:36:53.862 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:27:53.797 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:03.825 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:03.760 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:03.830 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:03.765 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:03.845 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:03.781 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:03.874 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:03.782 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:03.877 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:03.785 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:03.878 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:03.814 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:03.882 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:03.817 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:03.889 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:03.797 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:03.895 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:03.830 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:13.857 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:13.792 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:13.859 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:13.794 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:13.871 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:13.807 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:13.902 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:13.811 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:13.906 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:13.814 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:13.908 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:13.843 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:13.909 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:13.844 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:13.918 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:13.826 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:13.920 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:13.855 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:24.013 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:23.948 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:24.014 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:23.950 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:24.032 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:23.967 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:24.059 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:23.968 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:24.061 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:23.970 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:24.065 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:24.000 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:24.066 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:24.001 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:24.073 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:23.982 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:24.077 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:24.013 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:33.936 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:33.871 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:33.939 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:33.875 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:33.952 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:33.888 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:33.985 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:33.894 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:33.986 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:33.921 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:33.987 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:33.896 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:33.988 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:33.923 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:33.999 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:33.908 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:34.001 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:33.937 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:43.958 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:43.894 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:43.967 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:43.903 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:43.982 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:43.918 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:44.011 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:43.920 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:44.016 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:43.925 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:44.016 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:43.952 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:44.018 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:43.953 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:44.029 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:43.938 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:44.030 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:43.966 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:53.993 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:53.928 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:53.997 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:53.932 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:54.012 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:28:53.948 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:54.041 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:53.950 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:54.044 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:53.953 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:54.044 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:53.980 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:54.048 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:53.984 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:54.060 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:53.969 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:37:54.062 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:28:53.997 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:04.028 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:03.963 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:04.031 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:03.966 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:04.044 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:03.980 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:04.076 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:04.011 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:04.079 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:03.988 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:04.080 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:04.016 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:04.081 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:03.989 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:04.092 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:04.001 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:04.093 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:04.029 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:14.052 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:13.987 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:14.056 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:13.991 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:14.074 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:14.008 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:14.103 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:14.011 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:14.105 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:14.014 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:14.118 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:14.026 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:14.127 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:14.063 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:14.130 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:14.065 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:14.145 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:14.081 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:24.099 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:24.034 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:24.102 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:24.038 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:24.119 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:24.054 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:24.146 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:24.055 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:24.149 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:24.084 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:24.149 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:24.058 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:24.152 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:24.087 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:24.162 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:24.071 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:24.165 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:24.101 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:34.150 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:34.085 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:34.151 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:34.087 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:34.165 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:34.100 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:34.198 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:34.106 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:34.199 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:34.108 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:34.202 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:34.137 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:34.203 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:34.139 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:34.213 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:34.122 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:34.216 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:34.152 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:44.160 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:44.095 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:44.164 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:44.100 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:44.178 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:44.114 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:44.210 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:44.119 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:44.211 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:44.146 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:44.213 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:44.122 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:44.213 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:44.149 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:44.226 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:44.135 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:44.227 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:44.162 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:54.189 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:54.124 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:54.196 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:54.131 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:54.213 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:29:54.148 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:54.244 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:54.180 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:54.247 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:54.182 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:54.248 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:54.157 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:54.249 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:54.158 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:54.259 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:29:54.195 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:38:54.260 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:54.169 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:04.240 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:04.175 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:04.245 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:04.180 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:04.259 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:04.194 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:04.295 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:04.230 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:04.295 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:04.204 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:04.296 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:04.232 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:04.300 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:04.209 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:04.309 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:04.244 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:04.315 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:04.224 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:14.320 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:14.255 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:14.323 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:14.258 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:14.336 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:14.272 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:14.366 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:14.301 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:14.369 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:14.305 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:14.372 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:14.280 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:14.376 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:14.284 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:14.385 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:14.320 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:14.386 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:14.295 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:24.327 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:24.262 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:24.329 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:24.265 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:24.344 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:24.279 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:24.375 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:24.284 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:24.378 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:24.313 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:24.380 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:24.316 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:24.394 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:24.303 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:24.400 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:24.335 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:24.408 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:24.317 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:34.420 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:34.355 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:34.424 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:34.360 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:34.441 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:34.377 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:34.470 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:34.379 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:34.472 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:34.408 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:34.475 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:34.410 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:34.477 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:34.386 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:34.491 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:34.427 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:34.494 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:34.403 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:44.477 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:44.412 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:44.482 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:44.417 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:44.501 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:44.436 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:44.532 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:44.467 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:44.534 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:44.470 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:44.536 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:44.445 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:44.537 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:44.446 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:44.551 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:44.487 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:44.554 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:44.462 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:54.488 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:54.424 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:54.493 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:54.428 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:54.509 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:54.444 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:54.543 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:54.479 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:54.546 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:54.482 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:54.560 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:54.495 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:54.566 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:54.474 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:54.567 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:54.476 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:54.581 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:54.490 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:59.473 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:59.409 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:59.480 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:59.415 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:59.496 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:30:59.432 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:59.529 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:59.437 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:59.531 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:59.439 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:59.532 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:59.468 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:59.534 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:59.469 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:59.546 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:59.454 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:39:59.550 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:30:59.485 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:04.597 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:04.532 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:04.599 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:04.534 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:04.609 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:04.545 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:04.645 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:04.581 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:04.647 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:04.582 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:04.658 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:04.594 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:04.660 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:04.569 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:04.662 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:04.571 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:04.672 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:04.581 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:14.586 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:14.522 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:14.591 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:14.526 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:14.606 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:14.542 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:14.639 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:14.574 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:14.639 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:14.548 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:14.641 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:14.577 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:14.645 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:14.553 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:14.654 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:14.590 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:14.677 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:14.586 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:24.617 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:24.552 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:24.621 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:24.557 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:24.642 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:24.578 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:24.673 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:24.608 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:24.676 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:24.611 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:24.684 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:24.592 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:24.688 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:24.597 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:24.689 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:24.625 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:24.701 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:24.610 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:29.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-10-30T12:40:29.370 INFO:tasks.ceph.osd.4:Stopped 2023-10-30T12:40:29.371 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-30T12:40:29.371 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:40:29.383 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-10-30T12:40:29.374+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86362) UID: 0 2023-10-30T12:40:29.384 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-10-30T12:40:29.375+0000 ac7d640 -1 osd.5 910 *** Got signal Terminated *** 2023-10-30T12:40:34.681 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:34.616 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:34.684 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:34.619 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:34.700 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:34.635 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:34.733 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:34.668 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:34.735 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:34.671 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:34.747 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:34.656 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:34.748 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:34.657 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:34.749 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:34.685 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:34.763 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:34.672 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:34.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-10-30T12:40:34.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-10-30T12:40:35.474 INFO:tasks.ceph.osd.5:Stopped 2023-10-30T12:40:35.474 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-30T12:40:35.475 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:40:35.490 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-10-30T12:40:35.476+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86365) UID: 0 2023-10-30T12:40:35.490 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-10-30T12:40:35.477+0000 ac7d640 -1 osd.6 912 *** Got signal Terminated *** 2023-10-30T12:40:40.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-10-30T12:40:44.700 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:44.635 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:44.702 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:44.637 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:44.722 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:44.657 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:44.744 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:44.652 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:44.757 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:44.692 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:44.759 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:44.694 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:44.759 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:44.668 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:44.771 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:44.707 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:44.772 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:44.681 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:54.711 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:54.646 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:54.714 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:54.649 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:54.730 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:31:54.665 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:54.757 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:54.666 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:54.759 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:54.668 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:54.761 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:54.696 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:54.765 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:54.700 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:54.772 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:54.681 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:40:54.781 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:31:54.717 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:04.744 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:04.680 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:04.746 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:04.681 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:04.758 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:04.694 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:04.785 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:04.694 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:04.788 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:04.697 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:04.792 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:04.728 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:04.795 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:04.731 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:04.801 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:04.709 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:04.808 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:04.743 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:14.721 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:14.656 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:14.771 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:14.707 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:14.789 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:14.725 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:14.817 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:14.726 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:14.819 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:14.728 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:14.822 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:14.757 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:14.824 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:14.759 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:14.831 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:14.740 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:14.837 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:14.772 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:24.921 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:24.856 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:24.924 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:24.860 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:24.938 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:24.873 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:24.970 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:24.906 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:24.971 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:24.879 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:24.971 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:24.907 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:24.972 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:24.881 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:24.987 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:24.922 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:25.013 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:24.922 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:34.840 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:34.776 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:34.845 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:34.781 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:34.859 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:34.795 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:34.889 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:34.798 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:34.892 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:34.801 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:34.896 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:34.831 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:34.899 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:34.835 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:34.904 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:34.813 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:34.918 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:34.853 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:44.872 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:44.808 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:44.879 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:44.814 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:44.891 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:44.827 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:44.925 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:44.833 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:44.926 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:44.861 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:44.927 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:44.836 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:44.927 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:44.863 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:44.939 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:44.848 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:44.940 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:44.875 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:54.907 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:54.842 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:54.910 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:54.845 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:54.922 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:32:54.858 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:54.955 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:54.890 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:54.957 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:54.866 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:54.958 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:54.893 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:54.958 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:54.867 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:54.970 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:54.879 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:41:54.973 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:32:54.908 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:04.933 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:04.869 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:04.937 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:04.873 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:04.953 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:04.888 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:04.984 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:04.893 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:04.986 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:04.894 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:04.986 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:04.921 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:04.989 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:04.925 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:04.998 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:04.907 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:05.003 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:04.938 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:06.894 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-10-30T12:42:06.889+0000 f258640 -1 osd.7 912 heartbeat_check: no reply from 172.21.15.92:6811 osd.6 since back 2023-10-30T12:40:33.685869+0000 front 2023-10-30T12:40:33.687272+0000 (oldest deadline 2023-10-30T12:42:06.586909+0000) 2023-10-30T12:42:07.931 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-10-30T12:42:07.928+0000 f258640 -1 osd.7 912 heartbeat_check: no reply from 172.21.15.92:6811 osd.6 since back 2023-10-30T12:40:33.685869+0000 front 2023-10-30T12:40:33.687272+0000 (oldest deadline 2023-10-30T12:42:06.586909+0000) 2023-10-30T12:42:08.955 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-10-30T12:42:08.953+0000 f258640 -1 osd.7 912 heartbeat_check: no reply from 172.21.15.92:6811 osd.6 since back 2023-10-30T12:40:33.685869+0000 front 2023-10-30T12:40:33.687272+0000 (oldest deadline 2023-10-30T12:42:06.586909+0000) 2023-10-30T12:42:09.934 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-10-30T12:42:09.932+0000 f258640 -1 osd.7 912 heartbeat_check: no reply from 172.21.15.92:6811 osd.6 since back 2023-10-30T12:40:33.685869+0000 front 2023-10-30T12:40:33.687272+0000 (oldest deadline 2023-10-30T12:42:06.586909+0000) 2023-10-30T12:42:10.938 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-10-30T12:42:10.937+0000 f258640 -1 osd.7 912 heartbeat_check: no reply from 172.21.15.92:6811 osd.6 since back 2023-10-30T12:40:33.685869+0000 front 2023-10-30T12:40:33.687272+0000 (oldest deadline 2023-10-30T12:42:06.586909+0000) 2023-10-30T12:42:11.979 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-10-30T12:42:11.977+0000 f258640 -1 osd.7 912 heartbeat_check: no reply from 172.21.15.92:6811 osd.6 since back 2023-10-30T12:40:33.685869+0000 front 2023-10-30T12:40:33.687272+0000 (oldest deadline 2023-10-30T12:42:06.586909+0000) 2023-10-30T12:42:12.946 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-10-30T12:42:12.944+0000 f258640 -1 osd.7 912 heartbeat_check: no reply from 172.21.15.92:6811 osd.6 since back 2023-10-30T12:40:33.685869+0000 front 2023-10-30T12:40:33.687272+0000 (oldest deadline 2023-10-30T12:42:06.586909+0000) 2023-10-30T12:42:13.096 INFO:tasks.ceph.osd.6:Stopped 2023-10-30T12:42:13.097 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-30T12:42:13.097 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:42:13.112 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-10-30T12:42:13.103+0000 ac7d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86367) UID: 0 2023-10-30T12:42:13.112 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-10-30T12:42:13.104+0000 ac7d640 -1 osd.7 912 *** Got signal Terminated *** 2023-10-30T12:42:14.969 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:14.904 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:14.970 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:14.906 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:14.988 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:14.924 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:15.011 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:14.920 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:15.014 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:14.922 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:15.025 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:14.934 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:15.027 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:14.962 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:15.028 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:14.964 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:15.041 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:14.976 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:25.013 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:24.948 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:25.014 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:24.950 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:25.031 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:24.967 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:25.059 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:24.968 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:25.060 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:24.969 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:25.065 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:25.001 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:25.068 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:25.003 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:25.071 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:24.980 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:25.080 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:25.015 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:35.043 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:34.978 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:35.046 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:34.982 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:35.067 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:35.002 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:35.093 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:35.001 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:35.095 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:35.004 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:35.105 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:35.040 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:35.107 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:35.042 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:35.109 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:35.018 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:35.120 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:35.055 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:45.072 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:45.007 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:45.077 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:45.012 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:45.091 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:45.026 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:45.121 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:45.029 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:45.123 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:45.032 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:45.126 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:45.062 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:45.128 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:45.063 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:45.135 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:45.044 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:45.141 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:45.077 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:55.111 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:55.045 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:55.112 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:55.048 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:55.124 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:33:55.060 86439== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:55.156 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:55.065 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:55.158 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:55.067 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:55.159 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:55.094 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:55.160 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:55.096 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:55.171 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:55.080 85747== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:55.173 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:55.109 86438== Warning: unimplemented fcntl command: 1036 2023-10-30T12:42:55.808 INFO:tasks.ceph.osd.7:Stopped 2023-10-30T12:42:55.809 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-30T12:42:55.809 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-30T12:42:55.810 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:42:55.831 INFO:tasks.ceph.mgr.y:Stopped 2023-10-30T12:42:55.832 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-30T12:42:55.832 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:42:55.833 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:42:55.830+0000 7f67f6278640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 85745) UID: 0 2023-10-30T12:42:55.833 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-10-30T12:42:55.830+0000 7f67f6278640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-10-30T12:42:55.857 INFO:tasks.ceph.mgr.x:Stopped 2023-10-30T12:42:55.857 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-30T12:42:55.857 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:42:55.858 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:42:55.886 INFO:tasks.ceph.mon.a.smithi003.stderr:2023-10-30T12:42:55.874+0000 9f5f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 86401) UID: 0 2023-10-30T12:42:55.887 INFO:tasks.ceph.mon.a.smithi003.stderr:2023-10-30T12:42:55.875+0000 9f5f640 -1 mon.a@0(peon) e1 *** Got Signal Terminated *** 2023-10-30T12:42:56.592 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:56.528 86438== 2023-10-30T12:42:56.593 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:33:56.528 86438== Exit program on first error (--exit-on-first-error=yes) 2023-10-30T12:42:56.608 INFO:tasks.ceph.mon.a.smithi003.stderr:daemon-helper: command failed with exit status 42 2023-10-30T12:43:01.686 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-10-30T12:43:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-10-30T12:43:01.960 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-10-30T12:43:01.961 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:43:01.962 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T12:43:01.962 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:43:01.978 INFO:tasks.ceph.mon.c.smithi003.stderr:2023-10-30T12:43:01.965+0000 9f5f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 86403) UID: 0 2023-10-30T12:43:01.978 INFO:tasks.ceph.mon.c.smithi003.stderr:2023-10-30T12:43:01.967+0000 9f5f640 -1 mon.c@2(leader) e1 *** Got Signal Terminated *** 2023-10-30T12:43:05.476 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:34:05.411 86439== 2023-10-30T12:43:05.476 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:34:05.411 86439== Exit program on first error (--exit-on-first-error=yes) 2023-10-30T12:43:05.493 INFO:tasks.ceph.mon.c.smithi003.stderr:daemon-helper: command failed with exit status 42 2023-10-30T12:43:06.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-10-30T12:43:06.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-10-30T12:43:08.063 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-10-30T12:43:08.064 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-10-30T12:43:08.064 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T12:43:08.065 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T12:43:08.065 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:43:08.084 INFO:tasks.ceph.mon.b.smithi092.stderr:2023-10-30T12:43:08.068+0000 9f5f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 85743) UID: 0 2023-10-30T12:43:08.085 INFO:tasks.ceph.mon.b.smithi092.stderr:2023-10-30T12:43:08.069+0000 9f5f640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-10-30T12:43:08.832 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:08.741 85747== 2023-10-30T12:43:08.832 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:08.741 85747== Exit program on first error (--exit-on-first-error=yes) 2023-10-30T12:43:08.847 INFO:tasks.ceph.mon.b.smithi092.stderr:daemon-helper: command failed with exit status 42 2023-10-30T12:43:11.789 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-10-30T12:43:11.789 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-10-30T12:43:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-10-30T12:43:14.167 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi092 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-10-30T12:43:14.168 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T12:43:14.168 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-30T12:43:14.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-10-30T12:43:14.215 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi003.front.sepia.ceph.com 2023-10-30T12:43:14.216 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-30T12:43:14.358 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi003.front.sepia.ceph.com 2023-10-30T12:43:14.359 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-30T12:43:14.449 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi003.front.sepia.ceph.com 2023-10-30T12:43:14.450 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-30T12:43:14.516 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi003.front.sepia.ceph.com 2023-10-30T12:43:14.516 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-30T12:43:14.608 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi092.front.sepia.ceph.com 2023-10-30T12:43:14.608 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-30T12:43:14.677 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi092.front.sepia.ceph.com 2023-10-30T12:43:14.678 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-30T12:43:14.740 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi092.front.sepia.ceph.com 2023-10-30T12:43:14.741 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-30T12:43:14.815 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi092.front.sepia.ceph.com 2023-10-30T12:43:14.816 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-30T12:43:14.863 INFO:tasks.ceph:Archiving mon data... 2023-10-30T12:43:14.865 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070/data/mon.a.tgz 2023-10-30T12:43:14.866 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2023-10-30T12:43:14.882 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.RYcOYDVwVT 2023-10-30T12:43:14.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.RYcOYDVwVT 2023-10-30T12:43:15.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.RYcOYDVwVT 2023-10-30T12:43:15.138 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.RYcOYDVwVT is 426KB 2023-10-30T12:43:15.239 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.RYcOYDVwVT 2023-10-30T12:43:15.254 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070/data/mon.c.tgz 2023-10-30T12:43:15.254 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2023-10-30T12:43:15.309 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.ZuTyavso2J 2023-10-30T12:43:15.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.ZuTyavso2J 2023-10-30T12:43:15.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.ZuTyavso2J 2023-10-30T12:43:15.545 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.ZuTyavso2J is 426KB 2023-10-30T12:43:15.642 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.ZuTyavso2J 2023-10-30T12:43:15.657 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070/data/mon.b.tgz 2023-10-30T12:43:15.657 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2023-10-30T12:43:15.674 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.wY2cj9W0Jd 2023-10-30T12:43:15.675 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.wY2cj9W0Jd 2023-10-30T12:43:15.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.wY2cj9W0Jd 2023-10-30T12:43:15.931 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.wY2cj9W0Jd is 426KB 2023-10-30T12:43:16.027 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.wY2cj9W0Jd 2023-10-30T12:43:16.041 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-30T12:43:16.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T12:43:16.043 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T12:43:16.106 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-30T12:43:16.107 DEBUG:teuthology.orchestra.run.smithi003:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T12:43:16.115 DEBUG:teuthology.orchestra.run.smithi092:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T12:43:16.160 INFO:teuthology.orchestra.run.smithi003.stdout:/var/log/ceph/valgrind/mon.a.log: Leak_StillReachable 2023-10-30T12:43:16.160 INFO:teuthology.orchestra.run.smithi003.stdout:/var/log/ceph/valgrind/mon.c.log: Leak_StillReachable 2023-10-30T12:43:16.161 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind Leak_StillReachable 2023-10-30T12:43:16.161 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.a.log 2023-10-30T12:43:16.161 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind Leak_StillReachable 2023-10-30T12:43:16.162 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.c.log 2023-10-30T12:43:16.185 INFO:teuthology.orchestra.run.smithi092.stdout:/var/log/ceph/valgrind/mon.b.log: Leak_StillReachable 2023-10-30T12:43:16.185 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind Leak_StillReachable 2023-10-30T12:43:16.186 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.b.log 2023-10-30T12:43:16.186 INFO:tasks.ceph:Archiving crash dumps... 2023-10-30T12:43:16.188 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070/remote/smithi003/crash 2023-10-30T12:43:16.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T12:43:16.221 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070/remote/smithi092/crash 2023-10-30T12:43:16.222 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T12:43:16.251 INFO:tasks.ceph:Compressing logs... 2023-10-30T12:43:16.251 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T12:43:16.262 DEBUG:teuthology.orchestra.run.smithi092:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T12:43:16.289 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88563.log 2023-10-30T12:43:16.291 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89942.log 2023-10-30T12:43:16.292 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92332.log 2023-10-30T12:43:16.292 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.88563.log: /var/log/ceph/ceph-client.admin.89942.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-30T12:43:16.292 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92332.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88563.log.gz 2023-10-30T12:43:16.292 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86443.log 2023-10-30T12:43:16.292 INFO:teuthology.orchestra.run.smithi003.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.89942.log.gz 2023-10-30T12:43:16.293 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128525.log 2023-10-30T12:43:16.293 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.86443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92129.log 2023-10-30T12:43:16.293 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.128525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92367.log 2023-10-30T12:43:16.320 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158454.log 2023-10-30T12:43:16.320 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90674.log 2023-10-30T12:43:16.320 INFO:teuthology.orchestra.run.smithi003.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.86443.log.gz 2023-10-30T12:43:16.320 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90658.log 2023-10-30T12:43:16.320 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158202.log 2023-10-30T12:43:16.321 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.90658.log: /var/log/ceph/ceph-client.admin.90674.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.158454.log.gz 2023-10-30T12:43:16.321 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92677.log 2023-10-30T12:43:16.321 INFO:teuthology.orchestra.run.smithi003.stderr: 87.7%/var/log/ceph/ceph-client.admin.158202.log: -- replaced with /var/log/ceph/ceph-client.admin.90658.log.gz 2023-10-30T12:43:16.321 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91389.log 2023-10-30T12:43:16.321 INFO:teuthology.orchestra.run.smithi003.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.128525.log.gz 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90674.log.gz 2023-10-30T12:43:16.322 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:43:16.322 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92677.log: 88.6%gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2023-10-30T12:43:16.322 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.158202.log.gz 2023-10-30T12:43:16.322 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91389.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2023-10-30T12:43:16.322 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/valgrind/mon.c.log: gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91389.log.gz 2023-10-30T12:43:16.323 INFO:teuthology.orchestra.run.smithi003.stderr: -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2023-10-30T12:43:16.323 INFO:teuthology.orchestra.run.smithi003.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2023-10-30T12:43:16.323 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose --/var/log/ceph/valgrind/osd.2.log: /var/log/ceph/valgrind/osd.1.log 2023-10-30T12:43:16.323 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/valgrind/osd.0.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2023-10-30T12:43:16.323 INFO:teuthology.orchestra.run.smithi003.stderr:gzip 63.9% -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2023-10-30T12:43:16.323 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2023-10-30T12:43:16.324 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2023-10-30T12:43:16.324 INFO:teuthology.orchestra.run.smithi003.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2023-10-30T12:43:16.324 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90665.log 2023-10-30T12:43:16.324 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/valgrind/mon.a.log: /var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158664.log 2023-10-30T12:43:16.324 INFO:teuthology.orchestra.run.smithi003.stderr: 64.0% 74.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2023-10-30T12:43:16.325 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.90665.log: -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2023-10-30T12:43:16.325 INFO:teuthology.orchestra.run.smithi003.stderr:gzip 91.6% -- replaced with /var/log/ceph/ceph-client.admin.92332.log.gz 2023-10-30T12:43:16.325 INFO:teuthology.orchestra.run.smithi003.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.92259.log 2023-10-30T12:43:16.325 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158664.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90665.log.gz 2023-10-30T12:43:16.325 INFO:teuthology.orchestra.run.smithi003.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.92677.log.gz 2023-10-30T12:43:16.326 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157866.log 2023-10-30T12:43:16.326 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90671.log 2023-10-30T12:43:16.326 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92259.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.158664.log.gz 2023-10-30T12:43:16.326 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.157866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116307.log 2023-10-30T12:43:16.326 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.90671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128524.log 87.6% 2023-10-30T12:43:16.327 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.157866.log.gz 2023-10-30T12:43:16.327 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.116307.log: gzip -5 --verbose 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90671.log.gz 2023-10-30T12:43:16.327 INFO:teuthology.orchestra.run.smithi003.stderr: -- /var/log/ceph/ceph-client.admin.92485.log 2023-10-30T12:43:16.327 INFO:teuthology.orchestra.run.smithi003.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.92367.log.gz 2023-10-30T12:43:16.327 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.128524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92481.log 2023-10-30T12:43:16.328 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158580.log 2023-10-30T12:43:16.328 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92013.log 2023-10-30T12:43:16.328 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158580.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.158580.log.gz 2023-10-30T12:43:16.328 INFO:teuthology.orchestra.run.smithi003.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.116307.log.gz 2023-10-30T12:43:16.328 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138835.log 2023-10-30T12:43:16.329 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92013.log: gzip -5 --verbose -- 90.5% /var/log/ceph/ceph-client.admin.90204.log 2023-10-30T12:43:16.329 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.128524.log.gz 2023-10-30T12:43:16.329 INFO:teuthology.orchestra.run.smithi003.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.92259.log.gz 2023-10-30T12:43:16.329 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.138835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158370.log 2023-10-30T12:43:16.329 INFO:teuthology.orchestra.run.smithi003.stderr: 91.0%/var/log/ceph/ceph-client.admin.90204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92679.log 2023-10-30T12:43:16.330 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92485.log.gz 2023-10-30T12:43:16.330 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158412.log 2023-10-30T12:43:16.330 INFO:teuthology.orchestra.run.smithi003.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90204.log.gz 2023-10-30T12:43:16.330 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92679.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90450.log 2023-10-30T12:43:16.330 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.158370.log.gz 2023-10-30T12:43:16.331 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91000.log 2023-10-30T12:43:16.331 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.90450.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.92679.log.gz 2023-10-30T12:43:16.331 INFO:teuthology.orchestra.run.smithi003.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.158412.log.gzgzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80636.log 2023-10-30T12:43:16.331 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:43:16.331 INFO:teuthology.orchestra.run.smithi003.stderr: 87.6%/var/log/ceph/ceph-client.admin.91000.log: -- replaced with /var/log/ceph/ceph-client.admin.90450.log.gz 2023-10-30T12:43:16.332 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91246.log 2023-10-30T12:43:16.332 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.tmp-client.admin.80636.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T12:43:16.332 INFO:teuthology.orchestra.run.smithi003.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91000.log.gz 0.0% 2023-10-30T12:43:16.332 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.80636.log.gz 2023-10-30T12:43:16.332 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138837.log 2023-10-30T12:43:16.332 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158832.log 2023-10-30T12:43:16.333 INFO:teuthology.orchestra.run.smithi003.stderr: 88.1% 90.5% -- replaced with /var/log/ceph/ceph-client.admin.138835.log.gz 2023-10-30T12:43:16.333 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91246.log.gz 2023-10-30T12:43:16.333 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.138837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157950.log 2023-10-30T12:43:16.333 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158244.log 2023-10-30T12:43:16.333 INFO:teuthology.orchestra.run.smithi003.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.158832.log.gz 2023-10-30T12:43:16.334 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.157950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92226.log 2023-10-30T12:43:16.334 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158244.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.157950.log.gz 2023-10-30T12:43:16.334 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158538.log 2023-10-30T12:43:16.334 INFO:teuthology.orchestra.run.smithi003.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158244.log.gz 2023-10-30T12:43:16.334 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90669.log 90.6% -- replaced with /var/log/ceph/ceph-client.admin.138837.log.gz 2023-10-30T12:43:16.335 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:43:16.335 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90308.log 2023-10-30T12:43:16.335 INFO:teuthology.orchestra.run.smithi003.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.92481.log.gz 2023-10-30T12:43:16.335 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.90669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91960.log 2023-10-30T12:43:16.335 INFO:teuthology.orchestra.run.smithi003.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.158538.log.gz 2023-10-30T12:43:16.336 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.90308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91006.log 2023-10-30T12:43:16.336 INFO:teuthology.orchestra.run.smithi003.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90669.log.gz 2023-10-30T12:43:16.336 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91960.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90308.log.gz 2023-10-30T12:43:16.336 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86772.log 2023-10-30T12:43:16.336 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92399.log 2023-10-30T12:43:16.337 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.86772.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91006.log.gz 2023-10-30T12:43:16.337 INFO:teuthology.orchestra.run.smithi003.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86772.log.gz 2023-10-30T12:43:16.337 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91584.log 2023-10-30T12:43:16.337 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91419.log 2023-10-30T12:43:16.337 INFO:teuthology.orchestra.run.smithi003.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.91960.log.gz 2023-10-30T12:43:16.337 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86871.log 2023-10-30T12:43:16.338 INFO:teuthology.orchestra.run.smithi003.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91584.log.gz/var/log/ceph/ceph-client.admin.91419.log: 2023-10-30T12:43:16.338 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91875.log 2023-10-30T12:43:16.338 INFO:teuthology.orchestra.run.smithi003.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91419.log.gz 2023-10-30T12:43:16.338 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.86871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92671.log 2023-10-30T12:43:16.338 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91875.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86871.log.gz 2023-10-30T12:43:16.339 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108725.log 2023-10-30T12:43:16.339 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92671.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.92226.log.gz 2023-10-30T12:43:16.339 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91542.log 2023-10-30T12:43:16.354 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-30T12:43:16.354 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86021.log 2023-10-30T12:43:16.354 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85901.log 2023-10-30T12:43:16.355 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.86021.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2023-10-30T12:43:16.355 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.85901.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2023-10-30T12:43:16.355 INFO:teuthology.orchestra.run.smithi092.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86021.log.gz 2023-10-30T12:43:16.355 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2023-10-30T12:43:16.355 INFO:teuthology.orchestra.run.smithi092.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2023-10-30T12:43:16.356 INFO:teuthology.orchestra.run.smithi092.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.85901.log.gz 2023-10-30T12:43:16.356 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2023-10-30T12:43:16.356 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2023-10-30T12:43:16.356 INFO:teuthology.orchestra.run.smithi092.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2023-10-30T12:43:16.356 INFO:teuthology.orchestra.run.smithi092.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2023-10-30T12:43:16.357 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T12:43:16.357 INFO:teuthology.orchestra.run.smithi092.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2023-10-30T12:43:16.357 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85941.log 2023-10-30T12:43:16.357 INFO:teuthology.orchestra.run.smithi092.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2023-10-30T12:43:16.357 INFO:teuthology.orchestra.run.smithi092.stderr:gzip/var/log/ceph/ceph.audit.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.85981.log 2023-10-30T12:43:16.358 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.85941.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-30T12:43:16.358 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.85981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86112.log 2023-10-30T12:43:16.358 INFO:teuthology.orchestra.run.smithi092.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.85941.log.gz 2023-10-30T12:43:16.358 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-30T12:43:16.358 INFO:teuthology.orchestra.run.smithi092.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.85981.log.gz 2023-10-30T12:43:16.359 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.86112.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-30T12:43:16.359 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86062.log 2023-10-30T12:43:16.359 INFO:teuthology.orchestra.run.smithi092.stderr: 87.3%/var/log/ceph/ceph-osd.5.log: -- replaced with /var/log/ceph/ceph-client.admin.86112.log.gz 2023-10-30T12:43:16.359 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86252.log 2023-10-30T12:43:16.359 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.86062.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-30T12:43:16.360 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.86252.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.86062.log.gz 2023-10-30T12:43:16.360 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86152.log 2023-10-30T12:43:16.360 INFO:teuthology.orchestra.run.smithi092.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T12:43:16.360 INFO:teuthology.orchestra.run.smithi092.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86252.log.gz 2023-10-30T12:43:16.360 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.86152.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-30T12:43:16.361 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.86152.log.gz 2023-10-30T12:43:16.361 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mon.b.log: 92.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T12:43:16.361 INFO:teuthology.orchestra.run.smithi092.stderr: 87.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T12:43:16.376 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.108725.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.92671.log.gz 2023-10-30T12:43:16.376 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-30T12:43:16.376 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91542.log: 87.1% 91.3% -- replaced with /var/log/ceph/ceph-client.admin.91542.log.gz 2023-10-30T12:43:16.376 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92399.log.gz 2023-10-30T12:43:16.377 INFO:teuthology.orchestra.run.smithi003.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.108725.log.gz 2023-10-30T12:43:16.377 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90655.log 2023-10-30T12:43:16.377 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87575.log 2023-10-30T12:43:16.377 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.90655.log: 95.5% -- replaced with /var/log/ceph/ceph.audit.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158622.log 2023-10-30T12:43:16.377 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:43:16.378 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.87575.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90655.log.gz 2023-10-30T12:43:16.378 INFO:teuthology.orchestra.run.smithi003.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.92518.log 2023-10-30T12:43:16.378 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108724.log 2023-10-30T12:43:16.378 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92518.log: 87.7% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87575.log.gz 2023-10-30T12:43:16.378 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.158622.log.gz 2023-10-30T12:43:16.379 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91010.log 2023-10-30T12:43:16.379 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.108724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158328.log 2023-10-30T12:43:16.379 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91010.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91010.log.gz 2023-10-30T12:43:16.379 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116306.log 2023-10-30T12:43:16.379 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158328.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.108724.log.gz 2023-10-30T12:43:16.380 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90408.log 2023-10-30T12:43:16.380 INFO:teuthology.orchestra.run.smithi003.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.158328.log.gz 2023-10-30T12:43:16.380 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.116306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116314.log 2023-10-30T12:43:16.380 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.90408.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116306.log.gz 2023-10-30T12:43:16.380 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158496.log 2023-10-30T12:43:16.381 INFO:teuthology.orchestra.run.smithi003.stderr: 87.9%/var/log/ceph/ceph-client.admin.116314.log: -- replaced with /var/log/ceph/ceph-client.admin.90408.log.gz 2023-10-30T12:43:16.381 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157992.log 2023-10-30T12:43:16.381 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158496.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158496.log.gz 2023-10-30T12:43:16.381 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158748.log 2023-10-30T12:43:16.381 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.157992.log: 87.9% 90.8% -- replaced with /var/log/ceph/ceph-client.admin.116314.log.gz 2023-10-30T12:43:16.382 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.157992.log.gz 2023-10-30T12:43:16.382 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91286.log 2023-10-30T12:43:16.382 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90492.log 2023-10-30T12:43:16.382 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91286.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.158748.log.gz 2023-10-30T12:43:16.382 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92641.log 2023-10-30T12:43:16.383 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.90492.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91286.log.gz 2023-10-30T12:43:16.383 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121335.log 2023-10-30T12:43:16.383 INFO:teuthology.orchestra.run.smithi003.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90492.log.gz 2023-10-30T12:43:16.383 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157908.log 2023-10-30T12:43:16.383 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.121335.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.121335.log.gz 2023-10-30T12:43:16.383 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91326.log 2023-10-30T12:43:16.384 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.157908.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.92641.log.gz 2023-10-30T12:43:16.384 INFO:teuthology.orchestra.run.smithi003.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.157908.log.gzgzip 2023-10-30T12:43:16.384 INFO:teuthology.orchestra.run.smithi003.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.121334.log 2023-10-30T12:43:16.384 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90247.log 2023-10-30T12:43:16.384 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.121334.log: /var/log/ceph/ceph-client.admin.91326.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.91326.log.gz 2023-10-30T12:43:16.385 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158160.log 2023-10-30T12:43:16.385 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.90247.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90247.log.gz 2023-10-30T12:43:16.385 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92319.log 2023-10-30T12:43:16.385 INFO:teuthology.orchestra.run.smithi003.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.121334.log.gz 2023-10-30T12:43:16.385 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158160.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-30T12:43:16.385 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92319.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.158160.log.gz 2023-10-30T12:43:16.386 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116297.log 2023-10-30T12:43:16.386 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.3.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.92319.log.gz 2023-10-30T12:43:16.386 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91934.log 2023-10-30T12:43:16.386 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.116297.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.91875.log.gz 2023-10-30T12:43:16.386 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86745.log 2023-10-30T12:43:16.387 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91934.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.116297.log.gz 2023-10-30T12:43:16.387 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92233.log 2023-10-30T12:43:16.387 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.86745.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.86745.log.gz 2023-10-30T12:43:16.387 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92672.log 2023-10-30T12:43:16.387 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92233.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.91934.log.gz 2023-10-30T12:43:16.388 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158286.log 2023-10-30T12:43:16.388 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92672.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.92233.log.gz 2023-10-30T12:43:16.388 INFO:teuthology.orchestra.run.smithi003.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.92672.log.gz 2023-10-30T12:43:16.388 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92603.log 2023-10-30T12:43:16.388 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90668.log 2023-10-30T12:43:16.388 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92603.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.158286.log.gz 2023-10-30T12:43:16.389 INFO:teuthology.orchestra.run.smithi003.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.92603.log.gz 2023-10-30T12:43:16.389 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92536.log 2023-10-30T12:43:16.389 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.90668.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-30T12:43:16.389 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92536.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90668.log.gz 2023-10-30T12:43:16.389 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90995.log 2023-10-30T12:43:16.390 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mgr.y.log: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.92536.log.gz 2023-10-30T12:43:16.394 INFO:teuthology.orchestra.run.smithi003.stderr: 94.2% -- replaced with /var/log/ceph/ceph-client.admin.92129.log.gz 2023-10-30T12:43:16.394 INFO:teuthology.orchestra.run.smithi003.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.92518.log.gz 2023-10-30T12:43:16.394 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91896.log 2023-10-30T12:43:16.394 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.90995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92066.log 2023-10-30T12:43:16.395 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92123.log 2023-10-30T12:43:16.395 INFO:teuthology.orchestra.run.smithi003.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90995.log.gz 2023-10-30T12:43:16.395 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91206.log 2023-10-30T12:43:16.395 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92066.log: /var/log/ceph/ceph-client.admin.92123.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.92066.log.gz 2023-10-30T12:43:16.395 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138836.log 2023-10-30T12:43:16.396 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91206.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91206.log.gz 2023-10-30T12:43:16.396 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128526.log 2023-10-30T12:43:16.396 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.138836.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.138836.log.gz 2023-10-30T12:43:16.396 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91922.log 2023-10-30T12:43:16.396 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.128526.log: 94.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-30T12:43:16.397 INFO:teuthology.orchestra.run.smithi003.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.128526.log.gz 2023-10-30T12:43:16.397 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90356.log 2023-10-30T12:43:16.397 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158076.log 2023-10-30T12:43:16.397 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91922.log: /var/log/ceph/ceph-client.admin.90356.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.90356.log.gz 2023-10-30T12:43:16.401 INFO:teuthology.orchestra.run.smithi003.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.91922.log.gz 2023-10-30T12:43:16.402 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92591.log 2023-10-30T12:43:16.402 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91967.log 2023-10-30T12:43:16.403 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158076.log: /var/log/ceph/ceph-client.admin.92591.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158076.log.gz 2023-10-30T12:43:16.403 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91368.log 2023-10-30T12:43:16.404 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91967.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.92591.log.gz 2023-10-30T12:43:16.408 INFO:teuthology.orchestra.run.smithi003.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.91967.log.gz 2023-10-30T12:43:16.409 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91001.log 2023-10-30T12:43:16.409 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-30T12:43:16.410 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91368.log: /var/log/ceph/ceph-client.admin.91001.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91368.log.gz 2023-10-30T12:43:16.411 INFO:teuthology.orchestra.run.smithi003.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91001.log.gz 2023-10-30T12:43:16.411 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91009.log 2023-10-30T12:43:16.423 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T12:43:16.424 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91009.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.91009.log.gz 2023-10-30T12:43:16.436 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91003.log 2023-10-30T12:43:16.438 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.91896.log.gz 2023-10-30T12:43:16.448 INFO:teuthology.orchestra.run.smithi003.stderr: 93.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T12:43:16.449 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91500.log 2023-10-30T12:43:16.449 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5/var/log/ceph/ceph-client.admin.91003.log: --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-30T12:43:16.450 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91500.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91003.log.gz 2023-10-30T12:43:16.450 INFO:teuthology.orchestra.run.smithi003.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91500.log.gz 2023-10-30T12:43:16.456 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158118.log 2023-10-30T12:43:16.456 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90670.log 2023-10-30T12:43:16.457 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158118.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.158118.log.gz 2023-10-30T12:43:16.457 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92228.log 2023-10-30T12:43:16.458 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.90670.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90670.log.gz 2023-10-30T12:43:16.459 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121333.log 2023-10-30T12:43:16.469 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92228.log: 91.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158034.log 2023-10-30T12:43:16.473 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.121333.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.92228.log.gz 2023-10-30T12:43:16.473 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.121333.log.gz 2023-10-30T12:43:16.477 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108723.log 2023-10-30T12:43:16.478 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158034.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158034.log.gz 2023-10-30T12:43:16.479 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128522.log 2023-10-30T12:43:16.483 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.108723.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.108723.log.gz 2023-10-30T12:43:16.483 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128523.log 2023-10-30T12:43:16.487 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.128522.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.128522.log.gz 2023-10-30T12:43:16.487 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90998.log 2023-10-30T12:43:16.489 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.128523.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.128523.log.gz 2023-10-30T12:43:16.490 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158792.log 2023-10-30T12:43:16.491 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.90998.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90998.log.gz 2023-10-30T12:43:16.492 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116315.log 2023-10-30T12:43:16.493 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158792.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.158792.log.gz 2023-10-30T12:43:16.493 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91963.log 2023-10-30T12:43:16.494 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.116315.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.116315.log.gz 2023-10-30T12:43:16.495 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92403.log 2023-10-30T12:43:16.510 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.91963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158706.log 2023-10-30T12:43:16.520 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.92403.log: 91.5%gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-30T12:43:16.521 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92403.log.gz 2023-10-30T12:43:16.521 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.158706.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.158706.log.gz 2023-10-30T12:43:16.533 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92643.log 2023-10-30T12:43:16.558 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.92643.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.92643.log.gz 2023-10-30T12:43:16.578 INFO:teuthology.orchestra.run.smithi003.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.91963.log.gz 2023-10-30T12:43:16.693 INFO:teuthology.orchestra.run.smithi092.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-30T12:43:16.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-10-30T12:43:16.923 INFO:teuthology.orchestra.run.smithi003.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.admin.92123.log.gz 2023-10-30T12:43:17.691 INFO:teuthology.orchestra.run.smithi003.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92013.log.gz 2023-10-30T12:43:17.723 INFO:teuthology.orchestra.run.smithi092.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-30T12:43:17.904 INFO:teuthology.orchestra.run.smithi003.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-30T12:43:19.656 INFO:teuthology.orchestra.run.smithi003.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-30T12:43:19.731 INFO:teuthology.orchestra.run.smithi092.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-30T12:43:20.051 INFO:teuthology.orchestra.run.smithi003.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-30T12:43:20.079 INFO:teuthology.orchestra.run.smithi092.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-30T12:43:20.453 INFO:teuthology.orchestra.run.smithi092.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-30T12:43:20.538 INFO:teuthology.orchestra.run.smithi092.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-30T12:43:20.540 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-10-30T12:43:20.540 INFO:teuthology.orchestra.run.smithi092.stderr:real 0m4.227s 2023-10-30T12:43:20.541 INFO:teuthology.orchestra.run.smithi092.stderr:user 0m16.701s 2023-10-30T12:43:20.541 INFO:teuthology.orchestra.run.smithi092.stderr:sys 0m0.568s 2023-10-30T12:43:20.545 INFO:teuthology.orchestra.run.smithi003.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-30T12:43:20.736 INFO:teuthology.orchestra.run.smithi003.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-30T12:43:21.447 INFO:teuthology.orchestra.run.smithi003.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-30T12:43:21.448 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-30T12:43:21.449 INFO:teuthology.orchestra.run.smithi003.stderr:real 0m5.174s 2023-10-30T12:43:21.449 INFO:teuthology.orchestra.run.smithi003.stderr:user 0m23.736s 2023-10-30T12:43:21.449 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m0.768s 2023-10-30T12:43:21.449 INFO:tasks.ceph:Archiving logs... 2023-10-30T12:43:21.450 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070/remote/smithi003/log 2023-10-30T12:43:21.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T12:43:24.422 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070/remote/smithi092/log 2023-10-30T12:43:24.423 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T12:43:26.820 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 1957, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 368, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-10-30T12:43:26.821 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-30T12:43:26.830 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 1957, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d0db9f2f95b9d6344950185071cace2a50d296eb/qa/tasks/ceph.py", line 368, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-10-30T12:43:26.831 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-30T12:43:26.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T12:43:26.834 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T12:43:26.930 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-30T12:43:26.940 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-30T12:43:26.940 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T12:43:26.943 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T12:43:26.958 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2023-10-30T12:43:26.959 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2023-10-30T12:43:26.962 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T12:43:26.962 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2023-10-30T12:43:26.963 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 79 +2177us[+2244us] +/- 83ms 2023-10-30T12:43:26.963 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 -1755us[-1683us] +/- 56ms 2023-10-30T12:43:26.963 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 +339us[ +339us] +/- 76ms 2023-10-30T12:43:26.963 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T12:43:26.965 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T12:43:26.965 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2023-10-30T12:43:26.966 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 79 +2117us[+2117us] +/- 83ms 2023-10-30T12:43:26.966 INFO:teuthology.orchestra.run.smithi092.stdout:^* hv02.front.sepia.ceph.com 3 7 377 81 -1774us[-1750us] +/- 55ms 2023-10-30T12:43:26.966 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 +331us[ +331us] +/- 76ms 2023-10-30T12:43:26.966 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T12:43:26.967 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-30T12:43:26.996 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-30T12:43:26.997 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-30T12:43:27.006 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-30T12:43:27.115 DEBUG:teuthology.orchestra.run.smithi092:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-30T12:43:27.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T12:43:27.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:43:27.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T12:43:27.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:43:27.281 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-30T12:43:27.291 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-30T12:43:27.300 INFO:teuthology.task.internal:Duration was 3073.847950 seconds 2023-10-30T12:43:27.301 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-30T12:43:27.309 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-30T12:43:27.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T12:43:27.312 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T12:43:27.361 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T12:43:27.365 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T12:43:27.648 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-30T12:43:27.648 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2023-10-30T12:43:27.649 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T12:43:27.669 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2023-10-30T12:43:27.669 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T12:43:27.700 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-30T12:43:27.700 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T12:43:27.711 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T12:43:27.768 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-30T12:43:27.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T12:43:27.783 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T12:43:27.901 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-30T12:43:27.913 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-30T12:43:27.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T12:43:27.944 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T12:43:27.970 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-30T12:43:27.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:43:27.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:43:28.013 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2023-10-30T12:43:28.038 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2023-10-30T12:43:28.082 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:43:28.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:43:28.101 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:43:28.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:43:28.144 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-30T12:43:28.165 INFO:teuthology.task.internal:Transferring archived files... 2023-10-30T12:43:28.166 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070/remote/smithi003 2023-10-30T12:43:28.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T12:43:28.234 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070/remote/smithi092 2023-10-30T12:43:28.235 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T12:43:28.285 INFO:teuthology.task.internal:Removing archive directory... 2023-10-30T12:43:28.285 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T12:43:28.290 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T12:43:28.325 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-30T12:43:28.347 INFO:teuthology.task.internal:Not uploading archives. 2023-10-30T12:43:28.347 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-30T12:43:28.391 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-30T12:43:28.391 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T12:43:28.395 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T12:43:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: 265531 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 12:43 /home/ubuntu/cephtest 2023-10-30T12:43:28.409 INFO:teuthology.orchestra.run.smithi092.stdout: 265983 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 12:43 /home/ubuntu/cephtest 2023-10-30T12:43:28.410 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-30T12:43:28.421 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-30T12:43:28.432 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: saw valgrind issues 2023-10-30T12:43:28.433 INFO:teuthology.nuke:Checking targets against current locks 2023-10-30T12:43:28.455 DEBUG:teuthology.nuke:shortname: smithi003 2023-10-30T12:43:28.455 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T12:43:28.481 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 11:44:05.001677', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB0aWaga+dsDrTwqHzQWFfUjBQaami3eA1PWTEDHnqNfmchWzCWJ7rBcGnS6wHE8cDcDGGiFpluElQlxZf8rN+w='} 2023-10-30T12:43:28.483 DEBUG:teuthology.nuke:shortname: smithi092 2023-10-30T12:43:28.483 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T12:43:28.509 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2023-10-30_10:36:20-rados:verify-main-distro-default-smithi/7441070', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 11:44:05.003252', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBABPeEmKQ4kin+AOWQHfOC8gZ7lIz1qIaXii2x99f4NDEOM4MA9xORRfqdfj44euxLjUEZy/r5gm4sjfbgrWH7E='} 2023-10-30T12:43:28.539 INFO:teuthology.orchestra.console.smithi003:Power off 2023-10-30T12:43:28.540 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T12:43:28.563 DEBUG:teuthology.orchestra.console.smithi003:power off output: Chassis Power Control: Down/Off 2023-10-30T12:43:28.564 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:43:28.577 INFO:teuthology.orchestra.console.smithi092:Power off 2023-10-30T12:43:28.578 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T12:43:28.589 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2023-10-30T12:43:28.603 DEBUG:teuthology.orchestra.console.smithi092:power off output: Chassis Power Control: Down/Off 2023-10-30T12:43:28.603 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:43:28.626 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2023-10-30T12:43:32.591 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:43:32.627 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:43:32.702 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2023-10-30T12:43:32.738 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2023-10-30T12:43:36.707 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:43:36.739 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:43:36.818 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2023-10-30T12:43:36.849 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2023-10-30T12:43:40.822 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:43:40.850 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:43:40.932 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off 2023-10-30T12:43:40.960 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is off 2023-10-30T12:43:41.034 INFO:teuthology.orchestra.console.smithi003:Power off completed 2023-10-30T12:43:41.061 INFO:teuthology.orchestra.console.smithi092:Power off completed 2023-10-30T12:43:41.165 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind} duration: 3073.8479499816895 failure_reason: saw valgrind issues flavor: default owner: scheduled_amathuri@teuthology status: fail success: false 2023-10-30T12:43:41.165 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T12:43:41.235 INFO:teuthology.run:FAIL